[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:52 dist-test-slave-hc1k krb5kdc[12976](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:52 dist-test-slave-hc1k krb5kdc[12976](info): set up 2 sockets
Sep 26 13:44:52 dist-test-slave-hc1k krb5kdc[12976](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 13:44:52.908149 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:44:52 dist-test-slave-hc1k krb5kdc[12976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:44:53.025067 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:44921
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:38593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:44921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:53.129222 12989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:53.129444 12989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:53.129483 12989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:53.133527 12989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:53.133584 12989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:53.133605 12989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:53.133620 12989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:53.133632 12989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:53.138569 12989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:38593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:44921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:44921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:53.139412 12989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:53.140326 12989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:53.148399 12994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:53.148422 12997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:53.148531 12989 server_base.cc:1034] running on GCE node
W20260926 13:44:53.148383 12995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:53.149189 12989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:53.150189 12989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:53.151345 12989 hybrid_clock.cc:648] HybridClock initialized: now 1790430293151324 us; error 33 us; skew 500 ppm
Sep 26 13:44:53 dist-test-slave-hc1k krb5kdc[12976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430293, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:53.154070 12989 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:44:53.154877 12989 webserver.cc:460] Webserver started at http://127.12.170.126:40933/ using document root <none> and password file <none>
I20260926 13:44:53.155385 12989 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:53.155452 12989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:53.155682 12989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:53.157604 12989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "b5e871a42abc4172b7be789717692cf7"
format_stamp: "Formatted at 2026-09-26 13:44:53 on dist-test-slave-hc1k"
server_key: "2d14fcbb28f30a5430472f4a322b1485"
server_key_iv: "7ec98315ebb6f2907ad64dc56b668221"
server_key_version: "encryptionkey@0"
I20260926 13:44:53.158150 12989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "b5e871a42abc4172b7be789717692cf7"
format_stamp: "Formatted at 2026-09-26 13:44:53 on dist-test-slave-hc1k"
server_key: "2d14fcbb28f30a5430472f4a322b1485"
server_key_iv: "7ec98315ebb6f2907ad64dc56b668221"
server_key_version: "encryptionkey@0"
I20260926 13:44:53.161616 12989 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:44:53.163841 13004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:53.164916 12989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 13:44:53.165027 12989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "b5e871a42abc4172b7be789717692cf7"
format_stamp: "Formatted at 2026-09-26 13:44:53 on dist-test-slave-hc1k"
server_key: "2d14fcbb28f30a5430472f4a322b1485"
server_key_iv: "7ec98315ebb6f2907ad64dc56b668221"
server_key_version: "encryptionkey@0"
I20260926 13:44:53.165126 12989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:53.175233 12989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:53.177902 12989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:53.178079 12989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:53.184690 12989 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:44921
I20260926 13:44:53.184715 13056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:44921 every 8 connection(s)
I20260926 13:44:53.185539 12989 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:53.188067 13057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:53.189908 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 12989
I20260926 13:44:53.190030 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:44:53.191218 12969 external_mini_cluster.cc:1428] Setting key 073ed69102d9207e1a6d056018013eaf
I20260926 13:44:53.193197 13057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5e871a42abc4172b7be789717692cf7: Bootstrap starting.
I20260926 13:44:53.195807 13057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5e871a42abc4172b7be789717692cf7: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:53.196439 13057 log.cc:824] T 00000000000000000000000000000000 P b5e871a42abc4172b7be789717692cf7: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:53.198040 13057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5e871a42abc4172b7be789717692cf7: No bootstrap required, opened a new log
I20260926 13:44:53.201618 13057 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5e871a42abc4172b7be789717692cf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e871a42abc4172b7be789717692cf7" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 44921 } }
I20260926 13:44:53.201828 13057 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5e871a42abc4172b7be789717692cf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:53.201892 13057 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5e871a42abc4172b7be789717692cf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5e871a42abc4172b7be789717692cf7, State: Initialized, Role: FOLLOWER
I20260926 13:44:53.202257 13057 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5e871a42abc4172b7be789717692cf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e871a42abc4172b7be789717692cf7" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 44921 } }
I20260926 13:44:53.202374 13057 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5e871a42abc4172b7be789717692cf7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:53.202435 13057 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5e871a42abc4172b7be789717692cf7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:53.202522 13057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5e871a42abc4172b7be789717692cf7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:53.203285 13057 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5e871a42abc4172b7be789717692cf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e871a42abc4172b7be789717692cf7" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 44921 } }
I20260926 13:44:53.203580 13057 leader_election.cc:302] T 00000000000000000000000000000000 P b5e871a42abc4172b7be789717692cf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5e871a42abc4172b7be789717692cf7; no voters:
I20260926 13:44:53.203873 13057 leader_election.cc:288] T 00000000000000000000000000000000 P b5e871a42abc4172b7be789717692cf7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:53.203953 13062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5e871a42abc4172b7be789717692cf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:53.204119 13062 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5e871a42abc4172b7be789717692cf7 [term 1 LEADER]: Becoming Leader. State: Replica: b5e871a42abc4172b7be789717692cf7, State: Running, Role: LEADER
I20260926 13:44:53.204385 13062 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5e871a42abc4172b7be789717692cf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e871a42abc4172b7be789717692cf7" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 44921 } }
I20260926 13:44:53.205010 13057 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5e871a42abc4172b7be789717692cf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:53.207432 13064 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5e871a42abc4172b7be789717692cf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5e871a42abc4172b7be789717692cf7. Latest consensus state: current_term: 1 leader_uuid: "b5e871a42abc4172b7be789717692cf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e871a42abc4172b7be789717692cf7" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 44921 } } }
I20260926 13:44:53.207530 13064 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5e871a42abc4172b7be789717692cf7 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:53.207930 13063 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5e871a42abc4172b7be789717692cf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5e871a42abc4172b7be789717692cf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5e871a42abc4172b7be789717692cf7" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 44921 } } }
I20260926 13:44:53.207998 13063 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5e871a42abc4172b7be789717692cf7 [sys.catalog]: This master's current role is: LEADER
Sep 26 13:44:53 dist-test-slave-hc1k krb5kdc[12976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
W20260926 13:44:53.209689 13076 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5e871a42abc4172b7be789717692cf7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:44:53.209798 13076 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:44:53.209879 13078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:53.210697 13078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:53.214838 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:53.194121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59582 (local address 127.12.170.126:44921)
0926 13:44:53.194499 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:53.194514 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:53.195580 (+ 1066us) server_negotiation.cc:407] Connection header received
0926 13:44:53.198601 (+ 3021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:53.198638 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:53.199002 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:53.199334 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:53.201498 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:53.202639 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:53.203865 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:53.204543 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:53.204702 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:53.209200 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:53.209206 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:53.209221 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:53.209250 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:53.211371 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:53.212034 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:53.212040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:53.212045 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:53.212187 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:53.212589 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:53.212593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:53.212594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:53.213028 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:53.213200 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:53.213847 (+ 647us) server_negotiation.cc:299] Negotiation successful
0926 13:44:53.214098 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":112,"threads_started":1}
I20260926 13:44:53.217666 13078 catalog_manager.cc:1347] Generated new cluster ID: aa797b0fd717474282bd36bfda4112d3
I20260926 13:44:53.217721 13078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:53.227814 13078 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:53.228574 13078 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:53.232394 13078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5e871a42abc4172b7be789717692cf7: Generated new TSK 0
I20260926 13:44:53.232937 13078 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.170.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
I20260926 13:44:53.265956 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:0
--local_ip_for_outbound_sockets=127.12.170.65
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:44921
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:38593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:53.390790 13083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:53.391040 13083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:53.391083 13083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:53.395013 13083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:53.395069 13083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:53.395133 13083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:44:53.400215 13083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:38593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:44921
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:53.401185 13083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:53.402144 13083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:53.411087 13089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:53.411093 13088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:53.411259 13083 server_base.cc:1034] running on GCE node
W20260926 13:44:53.411093 13091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:53.411762 13083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:53.412307 13083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:53.413443 13083 hybrid_clock.cc:648] HybridClock initialized: now 1790430293413426 us; error 27 us; skew 500 ppm
Sep 26 13:44:53 dist-test-slave-hc1k krb5kdc[12976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430293, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:53.415913 13083 init.cc:377] Logged in from keytab as kudu/127.12.170.65@KRBTEST.COM (short username kudu)
I20260926 13:44:53.416659 13083 webserver.cc:460] Webserver started at http://127.12.170.65:40427/ using document root <none> and password file <none>
I20260926 13:44:53.417145 13083 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:53.417201 13083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:53.417378 13083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:53.419209 13083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-0/data/instance:
uuid: "648c495309a54dada21640e82f06fac3"
format_stamp: "Formatted at 2026-09-26 13:44:53 on dist-test-slave-hc1k"
server_key: "5b65edee0ebbdaddba00ab4f083a4c4e"
server_key_iv: "8574892913e55ab58d09227d51a5c37a"
server_key_version: "encryptionkey@0"
I20260926 13:44:53.419741 13083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance:
uuid: "648c495309a54dada21640e82f06fac3"
format_stamp: "Formatted at 2026-09-26 13:44:53 on dist-test-slave-hc1k"
server_key: "5b65edee0ebbdaddba00ab4f083a4c4e"
server_key_iv: "8574892913e55ab58d09227d51a5c37a"
server_key_version: "encryptionkey@0"
I20260926 13:44:53.423071 13083 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260926 13:44:53.425199 13098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:53.426277 13083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:44:53.426373 13083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "648c495309a54dada21640e82f06fac3"
format_stamp: "Formatted at 2026-09-26 13:44:53 on dist-test-slave-hc1k"
server_key: "5b65edee0ebbdaddba00ab4f083a4c4e"
server_key_iv: "8574892913e55ab58d09227d51a5c37a"
server_key_version: "encryptionkey@0"
I20260926 13:44:53.426460 13083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:53.449193 13083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:53.452065 13083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:53.452226 13083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:53.452730 13083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:44:53.453451 13083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:53.453495 13083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:53.453542 13083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:53.453568 13083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:53.461341 13083 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:36263
I20260926 13:44:53.461392 13211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:36263 every 8 connection(s)
I20260926 13:44:53.462181 13083 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
Sep 26 13:44:53 dist-test-slave-hc1k krb5kdc[12976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430293, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:44:53.471729 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 13083
I20260926 13:44:53.471827 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance
I20260926 13:44:53.472081 12969 external_mini_cluster.cc:1428] Setting key 714fc7c42491f0f7902a816522106664
I20260926 13:44:53.474535 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:53.463847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:59375 (local address 127.12.170.126:44921)
0926 13:44:53.463985 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:53.463989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:53.464897 (+ 908us) server_negotiation.cc:407] Connection header received
0926 13:44:53.465673 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:53.465676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:53.465722 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:53.465791 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:53.466913 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:53.467444 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:53.468702 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:53.468937 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:53.469023 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:53.471169 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:53.471172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:53.471174 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:53.471195 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:53.472575 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:53.473049 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:53.473052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:53.473054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:53.473137 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:53.473516 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:53.473518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:53.473519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:53.473698 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:53.473768 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:53.474335 (+ 567us) server_negotiation.cc:299] Negotiation successful
0926 13:44:53.474424 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 13:44:53.475455 13214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:53.464149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:44921 (local address 127.12.170.65:59375)
0926 13:44:53.464753 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:53.464792 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:53.465504 (+ 712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:53.465960 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:53.465966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:53.466309 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:53.466768 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:53.466781 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:53.467585 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:53.467589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:53.468574 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:53.468583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:53.469069 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:53.469074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:53.469240 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:53.469700 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:53.469715 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:53.471019 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:53.472679 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:53.472685 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:53.472698 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:53.472957 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:53.473227 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:53.473231 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:53.473233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:53.473394 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:53.473794 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:53.473800 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:53.474124 (+ 324us) client_negotiation.cc:769] Sending connection context
0926 13:44:53.474317 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 13:44:53.474613 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":106,"threads_started":1}
I20260926 13:44:53.476687 13212 heartbeater.cc:347] Connected to a master server at 127.12.170.126:44921
I20260926 13:44:53.476940 13212 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:53.477413 13212 heartbeater.cc:511] Master 127.12.170.126:44921 requested a full tablet report, sending...
I20260926 13:44:53.478983 13021 ts_manager.cc:194] Registered new tserver with Master: 648c495309a54dada21640e82f06fac3 (127.12.170.65:36263)
I20260926 13:44:53.480521 13021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.65@KRBTEST.COM'} at 127.12.170.65:59375
WARNING: no policy specified for kudu/127.12.170.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
I20260926 13:44:53.497835 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:0
--local_ip_for_outbound_sockets=127.12.170.66
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:44921
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:38593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:53.598166 13217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:53.598376 13217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:53.598414 13217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:53.602236 13217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:53.602283 13217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:53.602346 13217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:44:53.607283 13217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:38593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:44921
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:53.608155 13217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:53.609004 13217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:53.617431 13225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:53.617465 13222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:53.617457 13223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:53.617583 13217 server_base.cc:1034] running on GCE node
I20260926 13:44:53.618060 13217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:53.618608 13217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:53.619709 13217 hybrid_clock.cc:648] HybridClock initialized: now 1790430293619690 us; error 29 us; skew 500 ppm
Sep 26 13:44:53 dist-test-slave-hc1k krb5kdc[12976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430293, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:53.622143 13217 init.cc:377] Logged in from keytab as kudu/127.12.170.66@KRBTEST.COM (short username kudu)
I20260926 13:44:53.622886 13217 webserver.cc:460] Webserver started at http://127.12.170.66:33621/ using document root <none> and password file <none>
I20260926 13:44:53.623363 13217 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:53.623422 13217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:53.623585 13217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:53.625404 13217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-1/data/instance:
uuid: "7c20d4f9a82a457c9b1c39e924ab4da5"
format_stamp: "Formatted at 2026-09-26 13:44:53 on dist-test-slave-hc1k"
server_key: "cfbf3fe5dcf1f295b2eee80fd4f4c937"
server_key_iv: "3ebc21028781e224c1c4e7f8efb2c97d"
server_key_version: "encryptionkey@0"
I20260926 13:44:53.625890 13217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance:
uuid: "7c20d4f9a82a457c9b1c39e924ab4da5"
format_stamp: "Formatted at 2026-09-26 13:44:53 on dist-test-slave-hc1k"
server_key: "cfbf3fe5dcf1f295b2eee80fd4f4c937"
server_key_iv: "3ebc21028781e224c1c4e7f8efb2c97d"
server_key_version: "encryptionkey@0"
I20260926 13:44:53.629154 13217 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:44:53.631271 13232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:53.632364 13217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:44:53.632465 13217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "7c20d4f9a82a457c9b1c39e924ab4da5"
format_stamp: "Formatted at 2026-09-26 13:44:53 on dist-test-slave-hc1k"
server_key: "cfbf3fe5dcf1f295b2eee80fd4f4c937"
server_key_iv: "3ebc21028781e224c1c4e7f8efb2c97d"
server_key_version: "encryptionkey@0"
I20260926 13:44:53.632547 13217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:53.651705 13217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:53.654448 13217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:53.654601 13217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:53.655090 13217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:44:53.655853 13217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:53.655897 13217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:53.655946 13217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:53.655972 13217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:53.663867 13217 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:43259
I20260926 13:44:53.663928 13345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:43259 every 8 connection(s)
I20260926 13:44:53.664642 13217 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:44:53.672451 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 13217
I20260926 13:44:53.672559 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance
I20260926 13:44:53.672792 12969 external_mini_cluster.cc:1428] Setting key e59515cff6dbd8bf98c4c225fedee31d
Sep 26 13:44:53 dist-test-slave-hc1k krb5kdc[12976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430293, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:44:53.677084 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:53.666161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:36191 (local address 127.12.170.126:44921)
0926 13:44:53.666315 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:53.666319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:53.667293 (+ 974us) server_negotiation.cc:407] Connection header received
0926 13:44:53.668123 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:53.668126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:53.668169 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:53.668258 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:53.669339 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:53.669877 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:53.670932 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:53.671152 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:53.671236 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:53.673549 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:53.673552 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:53.673555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:53.673577 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:53.675055 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:53.675494 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:53.675497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:53.675499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:53.675610 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:53.675995 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:53.675998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:53.675999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:53.676180 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:53.676261 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:53.676837 (+ 576us) server_negotiation.cc:299] Negotiation successful
0926 13:44:53.676942 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 13:44:53.677775 13348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:53.666522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:44921 (local address 127.12.170.66:36191)
0926 13:44:53.667161 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:53.667199 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:53.667947 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:53.668437 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:53.668443 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:53.668778 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:53.669208 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:53.669221 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:53.669987 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:53.669990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:53.670819 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:53.670830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:53.671277 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:53.671281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:53.671431 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:53.671940 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:53.671959 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:53.673406 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:53.675164 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:53.675168 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:53.675178 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:53.675403 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:53.675744 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:53.675747 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:53.675750 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:53.675911 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:53.676270 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:53.676276 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:53.676547 (+ 271us) client_negotiation.cc:769] Sending connection context
0926 13:44:53.676756 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 13:44:53.677022 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":109,"threads_started":1}
I20260926 13:44:53.678761 13346 heartbeater.cc:347] Connected to a master server at 127.12.170.126:44921
I20260926 13:44:53.678985 13346 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:53.679492 13346 heartbeater.cc:511] Master 127.12.170.126:44921 requested a full tablet report, sending...
I20260926 13:44:53.680555 13021 ts_manager.cc:194] Registered new tserver with Master: 7c20d4f9a82a457c9b1c39e924ab4da5 (127.12.170.66:43259)
I20260926 13:44:53.681063 13021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.66@KRBTEST.COM'} at 127.12.170.66:36191
WARNING: no policy specified for kudu/127.12.170.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
I20260926 13:44:53.696769 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:0
--local_ip_for_outbound_sockets=127.12.170.67
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:44921
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:38593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:53.800791 13351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:53.801002 13351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:53.801041 13351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:53.804917 13351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:53.804962 13351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:53.805030 13351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:44:53.809994 13351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:38593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:44921
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:53.810818 13351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:53.811691 13351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:53.820065 13357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:53.820077 13359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:53.820173 13356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:53.820319 13351 server_base.cc:1034] running on GCE node
I20260926 13:44:53.820709 13351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:53.821261 13351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:53.822427 13351 hybrid_clock.cc:648] HybridClock initialized: now 1790430293822414 us; error 28 us; skew 500 ppm
Sep 26 13:44:53 dist-test-slave-hc1k krb5kdc[12976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430293, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:53.824867 13351 init.cc:377] Logged in from keytab as kudu/127.12.170.67@KRBTEST.COM (short username kudu)
I20260926 13:44:53.825591 13351 webserver.cc:460] Webserver started at http://127.12.170.67:44417/ using document root <none> and password file <none>
I20260926 13:44:53.826054 13351 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:53.826114 13351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:53.826288 13351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:53.828109 13351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-2/data/instance:
uuid: "661ca85cee624700a7ac6e3e2a8f7ddb"
format_stamp: "Formatted at 2026-09-26 13:44:53 on dist-test-slave-hc1k"
server_key: "803b19df1735856077829aea598eb64b"
server_key_iv: "0b013ed822dc26dfe61f092ba13faa0c"
server_key_version: "encryptionkey@0"
I20260926 13:44:53.828594 13351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance:
uuid: "661ca85cee624700a7ac6e3e2a8f7ddb"
format_stamp: "Formatted at 2026-09-26 13:44:53 on dist-test-slave-hc1k"
server_key: "803b19df1735856077829aea598eb64b"
server_key_iv: "0b013ed822dc26dfe61f092ba13faa0c"
server_key_version: "encryptionkey@0"
I20260926 13:44:53.831929 13351 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:44:53.833990 13366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:53.835098 13351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:44:53.835191 13351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "661ca85cee624700a7ac6e3e2a8f7ddb"
format_stamp: "Formatted at 2026-09-26 13:44:53 on dist-test-slave-hc1k"
server_key: "803b19df1735856077829aea598eb64b"
server_key_iv: "0b013ed822dc26dfe61f092ba13faa0c"
server_key_version: "encryptionkey@0"
I20260926 13:44:53.835270 13351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:53.847791 13351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:53.850060 13351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:53.850203 13351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:53.850673 13351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:44:53.851358 13351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:53.851399 13351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:53.851446 13351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:53.851473 13351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:53.858912 13351 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:36383
I20260926 13:44:53.858937 13479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:36383 every 8 connection(s)
I20260926 13:44:53.859674 13351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:44:53.861951 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 13351
I20260926 13:44:53.862047 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance
I20260926 13:44:53.862298 12969 external_mini_cluster.cc:1428] Setting key aa1133f53d1faf4a5da8b0c073a49c61
Sep 26 13:44:53 dist-test-slave-hc1k krb5kdc[12976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430293, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:44:53.871785 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:53.861222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:35551 (local address 127.12.170.126:44921)
0926 13:44:53.861342 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:53.861346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:53.862284 (+ 938us) server_negotiation.cc:407] Connection header received
0926 13:44:53.863139 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:53.863143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:53.863194 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:53.863266 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:53.864624 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:53.865170 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:53.866245 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:53.866457 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:53.866534 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:53.868885 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:53.868888 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:53.868890 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:53.868912 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:53.870117 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:53.870543 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:53.870545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:53.870546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:53.870621 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:53.870867 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:53.870869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:53.870870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:53.871027 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:53.871130 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:53.871552 (+ 422us) server_negotiation.cc:299] Negotiation successful
0926 13:44:53.871645 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:44:53.872494 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:53.861478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:44921 (local address 127.12.170.67:35551)
0926 13:44:53.862157 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:53.862199 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:53.862936 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:53.863429 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:53.863436 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:53.863908 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:53.864486 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:53.864502 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:53.865274 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:53.865277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:53.866138 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:53.866147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:53.866597 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:53.866602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:53.866765 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:53.867241 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:53.867257 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:53.868708 (+ 1451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:53.870214 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:53.870218 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:53.870228 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:53.870455 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:53.870685 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:53.870687 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:53.870688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:53.870801 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:53.871139 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:53.871144 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:53.871347 (+ 203us) client_negotiation.cc:769] Sending connection context
0926 13:44:53.871508 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 13:44:53.871786 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":100,"threads_started":1}
I20260926 13:44:53.873469 13480 heartbeater.cc:347] Connected to a master server at 127.12.170.126:44921
I20260926 13:44:53.873683 13480 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:53.874112 13480 heartbeater.cc:511] Master 127.12.170.126:44921 requested a full tablet report, sending...
I20260926 13:44:53.875030 13021 ts_manager.cc:194] Registered new tserver with Master: 661ca85cee624700a7ac6e3e2a8f7ddb (127.12.170.67:36383)
I20260926 13:44:53.875679 13021 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.67@KRBTEST.COM'} at 127.12.170.67:35551
I20260926 13:44:53.876365 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 13:44:53 dist-test-slave-hc1k krb5kdc[12976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430293, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 13:44:53 dist-test-slave-hc1k krb5kdc[12976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430293, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.170.65@KRBTEST.COM
I20260926 13:44:53.899456 13487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:53.891119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36204 (local address 127.12.170.65:36263)
0926 13:44:53.891427 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:53.891439 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:53.891461 (+ 22us) server_negotiation.cc:407] Connection header received
0926 13:44:53.891506 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:53.891511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:53.891716 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:53.891838 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:53.892479 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:53.893240 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:53.894028 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:53.894370 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:53.894459 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:53.896549 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:53.896558 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:53.896572 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:53.896594 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:53.897875 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:53.898253 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:53.898258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:53.898259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:53.898335 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:53.898565 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:53.898568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:53.898570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:53.898778 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:53.898951 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:53.899153 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 13:44:53.899305 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":113,"threads_started":1}
W20260926 13:44:53.900359 13126 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36204
Sep 26 13:44:53 dist-test-slave-hc1k krb5kdc[12976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 13:44:53 dist-test-slave-hc1k krb5kdc[12976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.65@KRBTEST.COM
I20260926 13:44:53.922662 13487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:53.915462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36220 (local address 127.12.170.65:36263)
0926 13:44:53.915574 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:53.915577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:53.915743 (+ 166us) server_negotiation.cc:407] Connection header received
0926 13:44:53.915867 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:53.915870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:53.915917 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:53.915982 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:53.916581 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:53.917073 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:53.917796 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:53.918006 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:53.918077 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:53.920078 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:53.920082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:53.920085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:53.920105 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:53.921289 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:53.921633 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:53.921637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:53.921638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:53.921704 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:53.921939 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:53.921942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:53.921944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:53.922089 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:53.922194 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:53.922407 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 13:44:53.922512 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 13:44:53.923475 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 13083
I20260926 13:44:53.927871 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 13217
I20260926 13:44:53.931813 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 13351
I20260926 13:44:53.935529 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 12989
2026-09-26T13:44:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1989 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:53 dist-test-slave-hc1k krb5kdc[13494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:53 dist-test-slave-hc1k krb5kdc[13494](info): set up 2 sockets
Sep 26 13:44:53 dist-test-slave-hc1k krb5kdc[13494](info): commencing operation
krb5kdc: starting...
W20260926 13:44:54.871191 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.916s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:44:54 dist-test-slave-hc1k krb5kdc[13494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:44:54.978418 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:40137
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:40649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:40137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:55.080240 13507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:55.080461 13507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:55.080513 13507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:55.084385 13507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:55.084430 13507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:55.084456 13507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:55.084478 13507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:55.084506 13507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:55.089370 13507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:40649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:40137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:40137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:55.090229 13507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:55.091099 13507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:55.098634 13512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:55.098692 13515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:55.098682 13513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:55.098914 13507 server_base.cc:1034] running on GCE node
I20260926 13:44:55.099416 13507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:55.100399 13507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:55.101538 13507 hybrid_clock.cc:648] HybridClock initialized: now 1790430295101520 us; error 37 us; skew 500 ppm
Sep 26 13:44:55 dist-test-slave-hc1k krb5kdc[13494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430295, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:55.104004 13507 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:44:55.104787 13507 webserver.cc:460] Webserver started at http://127.12.170.126:37885/ using document root <none> and password file <none>
I20260926 13:44:55.105271 13507 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:55.105330 13507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:55.105494 13507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:55.107285 13507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "2bd90206ada540e3b72e0bc8166e2eec"
format_stamp: "Formatted at 2026-09-26 13:44:55 on dist-test-slave-hc1k"
server_key: "5fa43e55a87300c0a286e7db7a59c8ce"
server_key_iv: "5d93f24379b511833e07e1f748d24a3f"
server_key_version: "encryptionkey@0"
I20260926 13:44:55.107800 13507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "2bd90206ada540e3b72e0bc8166e2eec"
format_stamp: "Formatted at 2026-09-26 13:44:55 on dist-test-slave-hc1k"
server_key: "5fa43e55a87300c0a286e7db7a59c8ce"
server_key_iv: "5d93f24379b511833e07e1f748d24a3f"
server_key_version: "encryptionkey@0"
I20260926 13:44:55.111025 13507 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:44:55.113255 13522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:55.114343 13507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:44:55.114451 13507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "2bd90206ada540e3b72e0bc8166e2eec"
format_stamp: "Formatted at 2026-09-26 13:44:55 on dist-test-slave-hc1k"
server_key: "5fa43e55a87300c0a286e7db7a59c8ce"
server_key_iv: "5d93f24379b511833e07e1f748d24a3f"
server_key_version: "encryptionkey@0"
I20260926 13:44:55.114540 13507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:55.132700 13507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:55.135628 13507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:55.135818 13507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:55.142204 13507 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:40137
I20260926 13:44:55.142246 13574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:40137 every 8 connection(s)
I20260926 13:44:55.143083 13507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:55.145681 13575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:55.150902 13575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bd90206ada540e3b72e0bc8166e2eec: Bootstrap starting.
I20260926 13:44:55.153120 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 13507
I20260926 13:44:55.153232 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:44:55.153342 13575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bd90206ada540e3b72e0bc8166e2eec: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:55.153507 12969 external_mini_cluster.cc:1428] Setting key 758e147f82592aea88accdf15073e2e4
I20260926 13:44:55.154127 13575 log.cc:824] T 00000000000000000000000000000000 P 2bd90206ada540e3b72e0bc8166e2eec: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:55.156095 13575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bd90206ada540e3b72e0bc8166e2eec: No bootstrap required, opened a new log
I20260926 13:44:55.159319 13575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bd90206ada540e3b72e0bc8166e2eec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd90206ada540e3b72e0bc8166e2eec" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40137 } }
I20260926 13:44:55.159518 13575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bd90206ada540e3b72e0bc8166e2eec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:55.159570 13575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bd90206ada540e3b72e0bc8166e2eec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bd90206ada540e3b72e0bc8166e2eec, State: Initialized, Role: FOLLOWER
I20260926 13:44:55.160045 13575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bd90206ada540e3b72e0bc8166e2eec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd90206ada540e3b72e0bc8166e2eec" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40137 } }
I20260926 13:44:55.160176 13575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bd90206ada540e3b72e0bc8166e2eec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:55.160238 13575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bd90206ada540e3b72e0bc8166e2eec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:55.160327 13575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bd90206ada540e3b72e0bc8166e2eec [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:55.161167 13575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bd90206ada540e3b72e0bc8166e2eec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd90206ada540e3b72e0bc8166e2eec" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40137 } }
Sep 26 13:44:55 dist-test-slave-hc1k krb5kdc[13494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:44:55.161506 13575 leader_election.cc:302] T 00000000000000000000000000000000 P 2bd90206ada540e3b72e0bc8166e2eec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bd90206ada540e3b72e0bc8166e2eec; no voters:
I20260926 13:44:55.161840 13575 leader_election.cc:288] T 00000000000000000000000000000000 P 2bd90206ada540e3b72e0bc8166e2eec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:55.161921 13580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bd90206ada540e3b72e0bc8166e2eec [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:55.162138 13580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bd90206ada540e3b72e0bc8166e2eec [term 1 LEADER]: Becoming Leader. State: Replica: 2bd90206ada540e3b72e0bc8166e2eec, State: Running, Role: LEADER
I20260926 13:44:55.162443 13580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bd90206ada540e3b72e0bc8166e2eec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd90206ada540e3b72e0bc8166e2eec" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40137 } }
I20260926 13:44:55.163102 13575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bd90206ada540e3b72e0bc8166e2eec [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:55.164059 13582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bd90206ada540e3b72e0bc8166e2eec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bd90206ada540e3b72e0bc8166e2eec. Latest consensus state: current_term: 1 leader_uuid: "2bd90206ada540e3b72e0bc8166e2eec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd90206ada540e3b72e0bc8166e2eec" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40137 } } }
I20260926 13:44:55.164187 13582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bd90206ada540e3b72e0bc8166e2eec [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:55.164463 13581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bd90206ada540e3b72e0bc8166e2eec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bd90206ada540e3b72e0bc8166e2eec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bd90206ada540e3b72e0bc8166e2eec" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40137 } } }
I20260926 13:44:55.164579 13581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bd90206ada540e3b72e0bc8166e2eec [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:55.167425 13579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:55.154644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35902 (local address 127.12.170.126:40137)
0926 13:44:55.154982 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:55.154996 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:55.155031 (+ 35us) server_negotiation.cc:407] Connection header received
0926 13:44:55.155634 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:55.155682 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:55.155998 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:55.156334 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:55.157061 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.157997 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.158808 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.159262 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.159399 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:55.162111 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:55.162117 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:55.162132 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:55.162160 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:55.164256 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:55.164876 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:55.164882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:55.164888 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:55.165026 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:55.165383 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:55.165387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:55.165389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:55.165880 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:55.166055 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:55.166520 (+ 465us) server_negotiation.cc:299] Negotiation successful
0926 13:44:55.166746 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":75,"threads_started":1}
W20260926 13:44:55.167685 13595 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2bd90206ada540e3b72e0bc8166e2eec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:44:55.167753 13595 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:44:55.171931 13589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:55.172683 13589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:55.177407 13589 catalog_manager.cc:1347] Generated new cluster ID: 773f387221d14ef8b5d9d823c3a6a303
I20260926 13:44:55.177461 13589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:55.190656 13589 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:55.191821 13589 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:55.202495 13589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bd90206ada540e3b72e0bc8166e2eec: Generated new TSK 0
I20260926 13:44:55.203177 13589 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.170.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
I20260926 13:44:55.232105 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:0
--local_ip_for_outbound_sockets=127.12.170.65
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:40137
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:40649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:55.333087 13601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:55.333302 13601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:55.333340 13601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:55.337208 13601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:55.337253 13601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:55.337312 13601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:44:55.342206 13601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:40649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:40137
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:55.343065 13601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:55.343972 13601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:55.352438 13606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:55.352432 13609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:55.352432 13607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:55.352666 13601 server_base.cc:1034] running on GCE node
I20260926 13:44:55.353046 13601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:55.353641 13601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:55.354797 13601 hybrid_clock.cc:648] HybridClock initialized: now 1790430295354777 us; error 33 us; skew 500 ppm
Sep 26 13:44:55 dist-test-slave-hc1k krb5kdc[13494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430295, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:55.357348 13601 init.cc:377] Logged in from keytab as kudu/127.12.170.65@KRBTEST.COM (short username kudu)
I20260926 13:44:55.358116 13601 webserver.cc:460] Webserver started at http://127.12.170.65:42353/ using document root <none> and password file <none>
I20260926 13:44:55.358594 13601 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:55.358647 13601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:55.358800 13601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:55.360700 13601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-0/data/instance:
uuid: "d8c8226497ca45aea3bcaefa65886732"
format_stamp: "Formatted at 2026-09-26 13:44:55 on dist-test-slave-hc1k"
server_key: "80d3c8c1f1e34d937f3cb082f3cef47a"
server_key_iv: "f99d6f0b5af3174eda253067d56ece27"
server_key_version: "encryptionkey@0"
I20260926 13:44:55.361172 13601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance:
uuid: "d8c8226497ca45aea3bcaefa65886732"
format_stamp: "Formatted at 2026-09-26 13:44:55 on dist-test-slave-hc1k"
server_key: "80d3c8c1f1e34d937f3cb082f3cef47a"
server_key_iv: "f99d6f0b5af3174eda253067d56ece27"
server_key_version: "encryptionkey@0"
I20260926 13:44:55.364596 13601 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:44:55.366801 13616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:55.367930 13601 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:44:55.368021 13601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "d8c8226497ca45aea3bcaefa65886732"
format_stamp: "Formatted at 2026-09-26 13:44:55 on dist-test-slave-hc1k"
server_key: "80d3c8c1f1e34d937f3cb082f3cef47a"
server_key_iv: "f99d6f0b5af3174eda253067d56ece27"
server_key_version: "encryptionkey@0"
I20260926 13:44:55.368106 13601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:55.378973 13601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:55.381625 13601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:55.381789 13601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:55.382233 13601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:44:55.382925 13601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:55.382966 13601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:55.383000 13601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:55.383019 13601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:55.390699 13601 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:46001
I20260926 13:44:55.390749 13729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:46001 every 8 connection(s)
I20260926 13:44:55.391496 13601 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:44:55.396540 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 13601
I20260926 13:44:55.396651 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance
I20260926 13:44:55.396911 12969 external_mini_cluster.cc:1428] Setting key aaf9e2ebdbc967b955169aa8d9e4de50
Sep 26 13:44:55 dist-test-slave-hc1k krb5kdc[13494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430295, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:44:55.404841 13579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:55.393155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:45967 (local address 127.12.170.126:40137)
0926 13:44:55.393304 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:55.393308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:55.394347 (+ 1039us) server_negotiation.cc:407] Connection header received
0926 13:44:55.395178 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:55.395180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:55.395223 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:55.395289 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:55.396480 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.397250 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.398453 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.398776 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.398878 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:55.401183 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:55.401186 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:55.401189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:55.401215 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:55.402764 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:55.403253 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:55.403256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:55.403258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:55.403345 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:55.403742 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:55.403746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:55.403748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:55.403955 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:55.404056 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:55.404563 (+ 507us) server_negotiation.cc:299] Negotiation successful
0926 13:44:55.404681 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.12.170.66@KRBTEST.COM; defaulting to no policy
I20260926 13:44:55.405493 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:55.393534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40137 (local address 127.12.170.65:45967)
0926 13:44:55.394208 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:55.394248 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:55.394978 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:55.395436 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:55.395443 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:55.395838 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:55.396350 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:55.396364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.397374 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.397377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:55.398341 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:55.398350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.398896 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.398898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:55.399023 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:55.399620 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:55.399636 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:55.401026 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:55.402892 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:55.402896 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:55.402907 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:55.403162 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:55.403474 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:55.403476 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:55.403478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:55.403612 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:55.404086 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:55.404090 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:55.404306 (+ 216us) client_negotiation.cc:769] Sending connection context
0926 13:44:55.404456 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 13:44:55.404708 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":108,"threads_started":1}
I20260926 13:44:55.406652 13730 heartbeater.cc:347] Connected to a master server at 127.12.170.126:40137
I20260926 13:44:55.406849 13730 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:55.407274 13730 heartbeater.cc:511] Master 127.12.170.126:40137 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.66@KRBTEST.COM" created.
I20260926 13:44:55.408658 13539 ts_manager.cc:194] Registered new tserver with Master: d8c8226497ca45aea3bcaefa65886732 (127.12.170.65:46001)
I20260926 13:44:55.409719 13539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.65@KRBTEST.COM'} at 127.12.170.65:45967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
I20260926 13:44:55.420146 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:0
--local_ip_for_outbound_sockets=127.12.170.66
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:40137
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:40649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:55.520272 13735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:55.520480 13735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:55.520519 13735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:55.524372 13735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:55.524415 13735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:55.524469 13735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:44:55.529320 13735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:40649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:40137
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:55.530153 13735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:55.530993 13735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:55.539314 13741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:55.539333 13740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:55.539315 13743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:55.539506 13735 server_base.cc:1034] running on GCE node
I20260926 13:44:55.539894 13735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:55.540412 13735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:55.541551 13735 hybrid_clock.cc:648] HybridClock initialized: now 1790430295541530 us; error 36 us; skew 500 ppm
Sep 26 13:44:55 dist-test-slave-hc1k krb5kdc[13494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430295, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:55.544113 13735 init.cc:377] Logged in from keytab as kudu/127.12.170.66@KRBTEST.COM (short username kudu)
I20260926 13:44:55.544828 13735 webserver.cc:460] Webserver started at http://127.12.170.66:36761/ using document root <none> and password file <none>
I20260926 13:44:55.545310 13735 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:55.545369 13735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:55.545540 13735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:55.547335 13735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-1/data/instance:
uuid: "75130b3f538642e998872357a82d46c0"
format_stamp: "Formatted at 2026-09-26 13:44:55 on dist-test-slave-hc1k"
server_key: "9cfcacc0ec0dd84ed651855f01452731"
server_key_iv: "8bc290dbe36dc793025d8bf94c4e2512"
server_key_version: "encryptionkey@0"
I20260926 13:44:55.547853 13735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance:
uuid: "75130b3f538642e998872357a82d46c0"
format_stamp: "Formatted at 2026-09-26 13:44:55 on dist-test-slave-hc1k"
server_key: "9cfcacc0ec0dd84ed651855f01452731"
server_key_iv: "8bc290dbe36dc793025d8bf94c4e2512"
server_key_version: "encryptionkey@0"
I20260926 13:44:55.551072 13735 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260926 13:44:55.553205 13750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:55.554260 13735 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:44:55.554352 13735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "75130b3f538642e998872357a82d46c0"
format_stamp: "Formatted at 2026-09-26 13:44:55 on dist-test-slave-hc1k"
server_key: "9cfcacc0ec0dd84ed651855f01452731"
server_key_iv: "8bc290dbe36dc793025d8bf94c4e2512"
server_key_version: "encryptionkey@0"
I20260926 13:44:55.554435 13735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:55.563547 13735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:55.565979 13735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:55.566134 13735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:55.566601 13735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:44:55.567293 13735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:55.567335 13735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:55.567380 13735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:55.567409 13735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:55.575196 13735 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:42689
I20260926 13:44:55.575224 13863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:42689 every 8 connection(s)
I20260926 13:44:55.575956 13735 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
Sep 26 13:44:55 dist-test-slave-hc1k krb5kdc[13494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430295, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:44:55.584606 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 13735
I20260926 13:44:55.584695 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance
I20260926 13:44:55.584916 12969 external_mini_cluster.cc:1428] Setting key b6d686eac627f264fc7baf752b6f0d1b
I20260926 13:44:55.587860 13579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:55.577461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:59219 (local address 127.12.170.126:40137)
0926 13:44:55.577588 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:55.577592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:55.578320 (+ 728us) server_negotiation.cc:407] Connection header received
0926 13:44:55.579150 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:55.579153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:55.579207 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:55.579276 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:55.580376 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.580892 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.581936 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.582184 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.582263 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:55.584374 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:55.584377 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:55.584379 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:55.584402 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:55.585828 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:55.586390 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:55.586394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:55.586396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:55.586478 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:55.586797 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:55.586800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:55.586802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:55.586981 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:55.587081 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:55.587540 (+ 459us) server_negotiation.cc:299] Negotiation successful
0926 13:44:55.587690 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 13:44:55.588591 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:55.577770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40137 (local address 127.12.170.66:59219)
0926 13:44:55.578195 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:55.578230 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:55.578961 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:55.579436 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:55.579443 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:55.579790 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:55.580234 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:55.580251 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.580989 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.580991 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:55.581831 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:55.581839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.582293 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.582296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:55.582423 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:55.582872 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:55.582887 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:55.584221 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:55.585944 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:55.585949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:55.585962 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:55.586293 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:55.586565 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:55.586568 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:55.586571 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:55.586706 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:55.587093 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:55.587099 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:55.587316 (+ 217us) client_negotiation.cc:769] Sending connection context
0926 13:44:55.587482 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 13:44:55.587812 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":110,"threads_started":1}
I20260926 13:44:55.589627 13864 heartbeater.cc:347] Connected to a master server at 127.12.170.126:40137
I20260926 13:44:55.589841 13864 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:55.590276 13864 heartbeater.cc:511] Master 127.12.170.126:40137 requested a full tablet report, sending...
I20260926 13:44:55.591218 13539 ts_manager.cc:194] Registered new tserver with Master: 75130b3f538642e998872357a82d46c0 (127.12.170.66:42689)
I20260926 13:44:55.591956 13539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.66@KRBTEST.COM'} at 127.12.170.66:59219
WARNING: no policy specified for kudu/127.12.170.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
I20260926 13:44:55.608268 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:0
--local_ip_for_outbound_sockets=127.12.170.67
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:40137
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:40649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:55.710395 13869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:55.710609 13869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:55.710649 13869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:55.714517 13869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:55.714567 13869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:55.714623 13869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:44:55.719606 13869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:40649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:40137
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:55.720424 13869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:55.721277 13869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:55.729754 13877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:55.729770 13875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:55.729770 13874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:55.729934 13869 server_base.cc:1034] running on GCE node
I20260926 13:44:55.730372 13869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:55.730893 13869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:55.732040 13869 hybrid_clock.cc:648] HybridClock initialized: now 1790430295732022 us; error 31 us; skew 500 ppm
Sep 26 13:44:55 dist-test-slave-hc1k krb5kdc[13494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430295, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:55.734422 13869 init.cc:377] Logged in from keytab as kudu/127.12.170.67@KRBTEST.COM (short username kudu)
I20260926 13:44:55.735150 13869 webserver.cc:460] Webserver started at http://127.12.170.67:40647/ using document root <none> and password file <none>
I20260926 13:44:55.735626 13869 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:55.735725 13869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:55.735899 13869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:55.737651 13869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-2/data/instance:
uuid: "451ca53e458745b4a103c32dd1d69035"
format_stamp: "Formatted at 2026-09-26 13:44:55 on dist-test-slave-hc1k"
server_key: "6475edb87852310658770a1c34ff7548"
server_key_iv: "401c012fdbf59b0b824f487ee57a5cc2"
server_key_version: "encryptionkey@0"
I20260926 13:44:55.738127 13869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance:
uuid: "451ca53e458745b4a103c32dd1d69035"
format_stamp: "Formatted at 2026-09-26 13:44:55 on dist-test-slave-hc1k"
server_key: "6475edb87852310658770a1c34ff7548"
server_key_iv: "401c012fdbf59b0b824f487ee57a5cc2"
server_key_version: "encryptionkey@0"
I20260926 13:44:55.741330 13869 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:44:55.743382 13884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:55.744470 13869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:44:55.744570 13869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "451ca53e458745b4a103c32dd1d69035"
format_stamp: "Formatted at 2026-09-26 13:44:55 on dist-test-slave-hc1k"
server_key: "6475edb87852310658770a1c34ff7548"
server_key_iv: "401c012fdbf59b0b824f487ee57a5cc2"
server_key_version: "encryptionkey@0"
I20260926 13:44:55.744649 13869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:55.762274 13869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:55.764899 13869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:55.765053 13869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:55.765517 13869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:44:55.766211 13869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:55.766253 13869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:55.766299 13869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:55.766328 13869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:55.774217 13869 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:41045
I20260926 13:44:55.774293 13997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:41045 every 8 connection(s)
I20260926 13:44:55.774977 13869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
Sep 26 13:44:55 dist-test-slave-hc1k krb5kdc[13494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430295, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:44:55.782953 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 13869
I20260926 13:44:55.783030 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance
I20260926 13:44:55.783282 12969 external_mini_cluster.cc:1428] Setting key 4e5fc79252781b2c725d20361ed55f62
I20260926 13:44:55.786924 13579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:55.776618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:57277 (local address 127.12.170.126:40137)
0926 13:44:55.776736 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:55.776739 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:55.777447 (+ 708us) server_negotiation.cc:407] Connection header received
0926 13:44:55.778252 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:55.778256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:55.778311 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:55.778382 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:55.779428 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.780002 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.781059 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.781274 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.781359 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:55.783618 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:55.783622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:55.783624 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:55.783674 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:55.785150 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:55.785627 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:55.785631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:55.785633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:55.785709 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:55.785997 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:55.786000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:55.786002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:55.786167 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:55.786247 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:55.786670 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 13:44:55.786777 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:44:55.787567 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:55.776902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40137 (local address 127.12.170.67:57277)
0926 13:44:55.777317 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:55.777354 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:55.778068 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:55.778544 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:55.778551 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:55.778853 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:55.779297 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:55.779309 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.780118 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.780121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:55.780947 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:55.780955 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.781398 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.781402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:55.781554 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:55.782001 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:55.782016 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:55.783453 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:55.785250 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:55.785255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:55.785267 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:55.785537 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:55.785785 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:55.785788 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:55.785790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:55.785917 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:55.786257 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:55.786263 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:55.786466 (+ 203us) client_negotiation.cc:769] Sending connection context
0926 13:44:55.786611 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 13:44:55.786884 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":115,"threads_started":1}
I20260926 13:44:55.788516 13998 heartbeater.cc:347] Connected to a master server at 127.12.170.126:40137
I20260926 13:44:55.788717 13998 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:55.789114 13998 heartbeater.cc:511] Master 127.12.170.126:40137 requested a full tablet report, sending...
I20260926 13:44:55.789983 13539 ts_manager.cc:194] Registered new tserver with Master: 451ca53e458745b4a103c32dd1d69035 (127.12.170.67:41045)
I20260926 13:44:55.790469 13539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.67@KRBTEST.COM'} at 127.12.170.67:57277
I20260926 13:44:55.796737 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:44:55.805788 13579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:55.799629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35904 (local address 127.12.170.126:40137)
0926 13:44:55.799750 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:55.799754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:55.799911 (+ 157us) server_negotiation.cc:407] Connection header received
0926 13:44:55.800020 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:55.800023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:55.800058 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:55.800116 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:55.800728 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.801192 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.801977 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.802177 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.802255 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:55.803374 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:55.803377 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:55.803379 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:55.803399 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:55.804394 (+ 995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:55.804741 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:55.804744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:55.804746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:55.804811 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:55.805036 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:55.805039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:55.805041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:55.805224 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:55.805333 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:55.805539 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 13:44:55.805661 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 13:44:55.809548 13539 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:44:55.812234 13539 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:44:55.824915 14014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:55.820457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44450 (local address 127.12.170.67:41045)
0926 13:44:55.820832 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:55.820842 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:55.820862 (+ 20us) server_negotiation.cc:407] Connection header received
0926 13:44:55.820924 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:55.820928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:55.821110 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:55.821236 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:55.821445 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.822153 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.823798 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.824341 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.824433 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:55.824570 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:55.824697 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 13:44:55.824781 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":121,"threads_started":1}
I20260926 13:44:55.825135 14011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:55.819807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:46001 (local address 127.0.0.1:52924)
0926 13:44:55.820025 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:55.820037 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:55.820113 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:55.821270 (+ 1157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:55.821272 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:55.821288 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:55.821397 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:55.821403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.822797 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.822800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:55.823781 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:55.823791 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.824775 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.824779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:55.824862 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:55.824886 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 13:44:55.824933 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 13:44:55.824996 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":21,"thread_start_us":92,"threads_started":1}
I20260926 13:44:55.825498 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:55.819445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:42689 (local address 127.0.0.1:57394)
0926 13:44:55.819902 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:55.819946 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:55.820068 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:55.820895 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:55.820900 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:55.820922 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:55.821047 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:55.821054 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.822589 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.822592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:55.823709 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:55.823715 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.825175 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.825179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:55.825259 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:55.825280 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 13:44:55.825326 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 13:44:55.825384 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":6,"thread_start_us":70,"threads_started":1}
I20260926 13:44:55.825673 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:55.820198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52924 (local address 127.12.170.65:46001)
0926 13:44:55.820851 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:55.820862 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:55.820882 (+ 20us) server_negotiation.cc:407] Connection header received
0926 13:44:55.820946 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:55.820950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:55.821134 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:55.821518 (+ 384us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:55.821673 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.822686 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.823873 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.824647 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.825277 (+ 630us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:55.825411 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:55.825456 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 13:44:55.825533 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":141,"threads_started":1}
I20260926 13:44:55.826189 14010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:55.819629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57394 (local address 127.12.170.66:42689)
0926 13:44:55.820421 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:55.820435 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:55.820456 (+ 21us) server_negotiation.cc:407] Connection header received
0926 13:44:55.820530 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:55.820535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:55.820732 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:55.821192 (+ 460us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:55.821347 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.822466 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.823846 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.825010 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.825785 (+ 775us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:55.825914 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:55.825956 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 13:44:55.826036 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":109,"threads_started":1}
I20260926 13:44:55.826409 14012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:55.820100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:41045 (local address 127.0.0.1:44450)
0926 13:44:55.820470 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:55.820481 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:55.820560 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:55.821224 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:55.821227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:55.821244 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:55.821355 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:55.821361 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.822263 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.822267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:55.823701 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:55.823711 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.824445 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.824448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:55.824545 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:55.824580 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 13:44:55.824672 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 13:44:55.825476 (+ 804us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":24,"thread_start_us":85,"threads_started":1}
I20260926 13:44:55.829396 13798 tablet_service.cc:1467] Processing CreateTablet for tablet 03c154b4fe90438caa33470cbb4e4475 (DEFAULT_TABLE table=test-table [id=19108f3302b340ec9ef0b761f5bfae70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:55.829396 13664 tablet_service.cc:1467] Processing CreateTablet for tablet 03c154b4fe90438caa33470cbb4e4475 (DEFAULT_TABLE table=test-table [id=19108f3302b340ec9ef0b761f5bfae70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:55.829741 13932 tablet_service.cc:1467] Processing CreateTablet for tablet 03c154b4fe90438caa33470cbb4e4475 (DEFAULT_TABLE table=test-table [id=19108f3302b340ec9ef0b761f5bfae70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:55.830451 13798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03c154b4fe90438caa33470cbb4e4475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:55.830451 13664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03c154b4fe90438caa33470cbb4e4475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:55.830701 13932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03c154b4fe90438caa33470cbb4e4475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:55.837258 14015 tablet_bootstrap.cc:492] T 03c154b4fe90438caa33470cbb4e4475 P 451ca53e458745b4a103c32dd1d69035: Bootstrap starting.
I20260926 13:44:55.837788 14017 tablet_bootstrap.cc:492] T 03c154b4fe90438caa33470cbb4e4475 P d8c8226497ca45aea3bcaefa65886732: Bootstrap starting.
I20260926 13:44:55.839141 14015 tablet_bootstrap.cc:654] T 03c154b4fe90438caa33470cbb4e4475 P 451ca53e458745b4a103c32dd1d69035: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:55.839699 14017 tablet_bootstrap.cc:654] T 03c154b4fe90438caa33470cbb4e4475 P d8c8226497ca45aea3bcaefa65886732: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:55.839980 14016 tablet_bootstrap.cc:492] T 03c154b4fe90438caa33470cbb4e4475 P 75130b3f538642e998872357a82d46c0: Bootstrap starting.
I20260926 13:44:55.840137 14015 log.cc:824] T 03c154b4fe90438caa33470cbb4e4475 P 451ca53e458745b4a103c32dd1d69035: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:55.840466 14017 log.cc:824] T 03c154b4fe90438caa33470cbb4e4475 P d8c8226497ca45aea3bcaefa65886732: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:55.842031 14016 tablet_bootstrap.cc:654] T 03c154b4fe90438caa33470cbb4e4475 P 75130b3f538642e998872357a82d46c0: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:55.842294 14015 tablet_bootstrap.cc:492] T 03c154b4fe90438caa33470cbb4e4475 P 451ca53e458745b4a103c32dd1d69035: No bootstrap required, opened a new log
I20260926 13:44:55.842294 14017 tablet_bootstrap.cc:492] T 03c154b4fe90438caa33470cbb4e4475 P d8c8226497ca45aea3bcaefa65886732: No bootstrap required, opened a new log
I20260926 13:44:55.842473 14017 ts_tablet_manager.cc:1397] T 03c154b4fe90438caa33470cbb4e4475 P d8c8226497ca45aea3bcaefa65886732: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 13:44:55.842473 14015 ts_tablet_manager.cc:1397] T 03c154b4fe90438caa33470cbb4e4475 P 451ca53e458745b4a103c32dd1d69035: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 13:44:55.842669 14016 log.cc:824] T 03c154b4fe90438caa33470cbb4e4475 P 75130b3f538642e998872357a82d46c0: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:55.844158 14016 tablet_bootstrap.cc:492] T 03c154b4fe90438caa33470cbb4e4475 P 75130b3f538642e998872357a82d46c0: No bootstrap required, opened a new log
I20260926 13:44:55.844321 14016 ts_tablet_manager.cc:1397] T 03c154b4fe90438caa33470cbb4e4475 P 75130b3f538642e998872357a82d46c0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 13:44:55.846131 14017 raft_consensus.cc:348] T 03c154b4fe90438caa33470cbb4e4475 P d8c8226497ca45aea3bcaefa65886732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75130b3f538642e998872357a82d46c0" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42689 } } peers { permanent_uuid: "451ca53e458745b4a103c32dd1d69035" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 41045 } } peers { permanent_uuid: "d8c8226497ca45aea3bcaefa65886732" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 46001 } }
I20260926 13:44:55.846131 14015 raft_consensus.cc:348] T 03c154b4fe90438caa33470cbb4e4475 P 451ca53e458745b4a103c32dd1d69035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75130b3f538642e998872357a82d46c0" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42689 } } peers { permanent_uuid: "451ca53e458745b4a103c32dd1d69035" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 41045 } } peers { permanent_uuid: "d8c8226497ca45aea3bcaefa65886732" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 46001 } }
I20260926 13:44:55.846392 14017 raft_consensus.cc:374] T 03c154b4fe90438caa33470cbb4e4475 P d8c8226497ca45aea3bcaefa65886732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:55.846392 14015 raft_consensus.cc:374] T 03c154b4fe90438caa33470cbb4e4475 P 451ca53e458745b4a103c32dd1d69035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:55.846446 14015 raft_consensus.cc:729] T 03c154b4fe90438caa33470cbb4e4475 P 451ca53e458745b4a103c32dd1d69035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 451ca53e458745b4a103c32dd1d69035, State: Initialized, Role: FOLLOWER
I20260926 13:44:55.846446 14017 raft_consensus.cc:729] T 03c154b4fe90438caa33470cbb4e4475 P d8c8226497ca45aea3bcaefa65886732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8c8226497ca45aea3bcaefa65886732, State: Initialized, Role: FOLLOWER
I20260926 13:44:55.846869 14017 consensus_queue.cc:260] T 03c154b4fe90438caa33470cbb4e4475 P d8c8226497ca45aea3bcaefa65886732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75130b3f538642e998872357a82d46c0" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42689 } } peers { permanent_uuid: "451ca53e458745b4a103c32dd1d69035" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 41045 } } peers { permanent_uuid: "d8c8226497ca45aea3bcaefa65886732" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 46001 } }
I20260926 13:44:55.847076 14016 raft_consensus.cc:348] T 03c154b4fe90438caa33470cbb4e4475 P 75130b3f538642e998872357a82d46c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75130b3f538642e998872357a82d46c0" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42689 } } peers { permanent_uuid: "451ca53e458745b4a103c32dd1d69035" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 41045 } } peers { permanent_uuid: "d8c8226497ca45aea3bcaefa65886732" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 46001 } }
I20260926 13:44:55.847214 14015 consensus_queue.cc:260] T 03c154b4fe90438caa33470cbb4e4475 P 451ca53e458745b4a103c32dd1d69035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75130b3f538642e998872357a82d46c0" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42689 } } peers { permanent_uuid: "451ca53e458745b4a103c32dd1d69035" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 41045 } } peers { permanent_uuid: "d8c8226497ca45aea3bcaefa65886732" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 46001 } }
I20260926 13:44:55.847323 14016 raft_consensus.cc:374] T 03c154b4fe90438caa33470cbb4e4475 P 75130b3f538642e998872357a82d46c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:55.847378 14016 raft_consensus.cc:729] T 03c154b4fe90438caa33470cbb4e4475 P 75130b3f538642e998872357a82d46c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75130b3f538642e998872357a82d46c0, State: Initialized, Role: FOLLOWER
I20260926 13:44:55.847776 14016 consensus_queue.cc:260] T 03c154b4fe90438caa33470cbb4e4475 P 75130b3f538642e998872357a82d46c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75130b3f538642e998872357a82d46c0" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42689 } } peers { permanent_uuid: "451ca53e458745b4a103c32dd1d69035" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 41045 } } peers { permanent_uuid: "d8c8226497ca45aea3bcaefa65886732" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 46001 } }
I20260926 13:44:55.847891 14017 ts_tablet_manager.cc:1428] T 03c154b4fe90438caa33470cbb4e4475 P d8c8226497ca45aea3bcaefa65886732: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 13:44:55.848115 14015 ts_tablet_manager.cc:1428] T 03c154b4fe90438caa33470cbb4e4475 P 451ca53e458745b4a103c32dd1d69035: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 13:44:55.848218 13730 heartbeater.cc:503] Master 127.12.170.126:40137 was elected leader, sending a full tablet report...
I20260926 13:44:55.848352 13864 heartbeater.cc:503] Master 127.12.170.126:40137 was elected leader, sending a full tablet report...
I20260926 13:44:55.848361 13998 heartbeater.cc:503] Master 127.12.170.126:40137 was elected leader, sending a full tablet report...
I20260926 13:44:55.849473 14016 ts_tablet_manager.cc:1428] T 03c154b4fe90438caa33470cbb4e4475 P 75130b3f538642e998872357a82d46c0: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260926 13:44:55.878952 14023 raft_consensus.cc:482] T 03c154b4fe90438caa33470cbb4e4475 P 75130b3f538642e998872357a82d46c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:44:55.879133 14023 raft_consensus.cc:504] T 03c154b4fe90438caa33470cbb4e4475 P 75130b3f538642e998872357a82d46c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75130b3f538642e998872357a82d46c0" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42689 } } peers { permanent_uuid: "451ca53e458745b4a103c32dd1d69035" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 41045 } } peers { permanent_uuid: "d8c8226497ca45aea3bcaefa65886732" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 46001 } }
I20260926 13:44:55.880167 14023 leader_election.cc:288] T 03c154b4fe90438caa33470cbb4e4475 P 75130b3f538642e998872357a82d46c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 451ca53e458745b4a103c32dd1d69035 (127.12.170.67:41045), d8c8226497ca45aea3bcaefa65886732 (127.12.170.65:46001)
I20260926 13:44:55.883313 13866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:55.880335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:41045 (local address 127.12.170.66:53159)
0926 13:44:55.880505 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:55.880519 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:55.880614 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:55.880832 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:55.880836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:55.880852 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:55.880951 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:55.880956 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.881614 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.881618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:55.882369 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:55.882376 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.882950 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.882953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:55.883049 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:55.883066 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 13:44:55.883128 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 13:44:55.883194 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 13:44:55.883313 14014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:55.880432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:53159 (local address 127.12.170.67:41045)
0926 13:44:55.880554 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:55.880559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:55.880573 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:44:55.880709 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:55.880711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:55.880746 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:55.880810 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:55.881048 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.881522 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.882477 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.882856 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.882949 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:55.882997 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:55.883147 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 13:44:55.883206 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 13:44:55.883535 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:55.880450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:40475 (local address 127.12.170.65:46001)
0926 13:44:55.880547 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:55.880551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:55.880787 (+ 236us) server_negotiation.cc:407] Connection header received
0926 13:44:55.880881 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:55.880884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:55.880923 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:55.880982 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:55.881214 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.881689 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.882672 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.883057 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.883158 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:55.883203 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:55.883350 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 13:44:55.883421 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 13:44:55.883715 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:55.880371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:46001 (local address 127.12.170.66:40475)
0926 13:44:55.880713 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:55.880723 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:55.880798 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:55.880998 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:55.881001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:55.881014 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:55.881122 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:55.881127 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.881796 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.881800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:55.882560 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:55.882567 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.883161 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.883164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:55.883253 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:55.883271 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 13:44:55.883330 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 13:44:55.883504 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":105,"threads_started":1}
I20260926 13:44:55.883955 13952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c154b4fe90438caa33470cbb4e4475" candidate_uuid: "75130b3f538642e998872357a82d46c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451ca53e458745b4a103c32dd1d69035" is_pre_election: true
I20260926 13:44:55.884213 13952 raft_consensus.cc:2454] T 03c154b4fe90438caa33470cbb4e4475 P 451ca53e458745b4a103c32dd1d69035 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75130b3f538642e998872357a82d46c0 in term 0.
I20260926 13:44:55.884238 13684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c154b4fe90438caa33470cbb4e4475" candidate_uuid: "75130b3f538642e998872357a82d46c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8c8226497ca45aea3bcaefa65886732" is_pre_election: true
I20260926 13:44:55.884488 13684 raft_consensus.cc:2454] T 03c154b4fe90438caa33470cbb4e4475 P d8c8226497ca45aea3bcaefa65886732 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75130b3f538642e998872357a82d46c0 in term 0.
I20260926 13:44:55.884670 13751 leader_election.cc:302] T 03c154b4fe90438caa33470cbb4e4475 P 75130b3f538642e998872357a82d46c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 451ca53e458745b4a103c32dd1d69035, 75130b3f538642e998872357a82d46c0; no voters:
I20260926 13:44:55.884902 14023 raft_consensus.cc:2781] T 03c154b4fe90438caa33470cbb4e4475 P 75130b3f538642e998872357a82d46c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:44:55.884979 14023 raft_consensus.cc:482] T 03c154b4fe90438caa33470cbb4e4475 P 75130b3f538642e998872357a82d46c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:44:55.885025 14023 raft_consensus.cc:3037] T 03c154b4fe90438caa33470cbb4e4475 P 75130b3f538642e998872357a82d46c0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:55.885958 14023 raft_consensus.cc:504] T 03c154b4fe90438caa33470cbb4e4475 P 75130b3f538642e998872357a82d46c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75130b3f538642e998872357a82d46c0" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42689 } } peers { permanent_uuid: "451ca53e458745b4a103c32dd1d69035" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 41045 } } peers { permanent_uuid: "d8c8226497ca45aea3bcaefa65886732" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 46001 } }
I20260926 13:44:55.886231 14023 leader_election.cc:288] T 03c154b4fe90438caa33470cbb4e4475 P 75130b3f538642e998872357a82d46c0 [CANDIDATE]: Term 1 election: Requested vote from peers 451ca53e458745b4a103c32dd1d69035 (127.12.170.67:41045), d8c8226497ca45aea3bcaefa65886732 (127.12.170.65:46001)
I20260926 13:44:55.886515 13952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c154b4fe90438caa33470cbb4e4475" candidate_uuid: "75130b3f538642e998872357a82d46c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "451ca53e458745b4a103c32dd1d69035"
I20260926 13:44:55.886541 13684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03c154b4fe90438caa33470cbb4e4475" candidate_uuid: "75130b3f538642e998872357a82d46c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8c8226497ca45aea3bcaefa65886732"
I20260926 13:44:55.886623 13684 raft_consensus.cc:3037] T 03c154b4fe90438caa33470cbb4e4475 P d8c8226497ca45aea3bcaefa65886732 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:55.886622 13952 raft_consensus.cc:3037] T 03c154b4fe90438caa33470cbb4e4475 P 451ca53e458745b4a103c32dd1d69035 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:55.887539 13684 raft_consensus.cc:2454] T 03c154b4fe90438caa33470cbb4e4475 P d8c8226497ca45aea3bcaefa65886732 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75130b3f538642e998872357a82d46c0 in term 1.
I20260926 13:44:55.887570 13952 raft_consensus.cc:2454] T 03c154b4fe90438caa33470cbb4e4475 P 451ca53e458745b4a103c32dd1d69035 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75130b3f538642e998872357a82d46c0 in term 1.
I20260926 13:44:55.887841 13753 leader_election.cc:302] T 03c154b4fe90438caa33470cbb4e4475 P 75130b3f538642e998872357a82d46c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75130b3f538642e998872357a82d46c0, d8c8226497ca45aea3bcaefa65886732; no voters:
I20260926 13:44:55.888033 14023 raft_consensus.cc:2781] T 03c154b4fe90438caa33470cbb4e4475 P 75130b3f538642e998872357a82d46c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:55.888306 14023 raft_consensus.cc:686] T 03c154b4fe90438caa33470cbb4e4475 P 75130b3f538642e998872357a82d46c0 [term 1 LEADER]: Becoming Leader. State: Replica: 75130b3f538642e998872357a82d46c0, State: Running, Role: LEADER
I20260926 13:44:55.888643 14023 consensus_queue.cc:237] T 03c154b4fe90438caa33470cbb4e4475 P 75130b3f538642e998872357a82d46c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75130b3f538642e998872357a82d46c0" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42689 } } peers { permanent_uuid: "451ca53e458745b4a103c32dd1d69035" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 41045 } } peers { permanent_uuid: "d8c8226497ca45aea3bcaefa65886732" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 46001 } }
I20260926 13:44:55.891485 13538 catalog_manager.cc:5495] T 03c154b4fe90438caa33470cbb4e4475 P 75130b3f538642e998872357a82d46c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 75130b3f538642e998872357a82d46c0 (127.12.170.66). New cstate: current_term: 1 leader_uuid: "75130b3f538642e998872357a82d46c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75130b3f538642e998872357a82d46c0" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "451ca53e458745b4a103c32dd1d69035" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 41045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8c8226497ca45aea3bcaefa65886732" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 46001 } health_report { overall_health: UNKNOWN } } }
W20260926 13:44:55.892526 13731 tablet.cc:2369] T 03c154b4fe90438caa33470cbb4e4475 P d8c8226497ca45aea3bcaefa65886732: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 13:44:55 dist-test-slave-hc1k krb5kdc[13494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.65@KRBTEST.COM
I20260926 13:44:55.909257 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:55.901263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52934 (local address 127.12.170.65:46001)
0926 13:44:55.901404 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:55.901408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:55.901539 (+ 131us) server_negotiation.cc:407] Connection header received
0926 13:44:55.901645 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:55.901648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:55.901688 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:55.901751 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:55.902370 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.902875 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.903755 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.903972 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.904056 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:55.906441 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:55.906451 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:55.906466 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:55.906495 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:55.907901 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:55.908267 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:55.908272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:55.908273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:55.908354 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:55.908533 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:55.908535 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:55.908536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:55.908761 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:55.908900 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:55.909024 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 13:44:55.909112 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 13:44:55 dist-test-slave-hc1k krb5kdc[13494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430295, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 13:44:55 dist-test-slave-hc1k krb5kdc[13494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430295, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.170.65@KRBTEST.COM
I20260926 13:44:55.932873 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:55.925472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52944 (local address 127.12.170.65:46001)
0926 13:44:55.925607 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:55.925610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:55.925713 (+ 103us) server_negotiation.cc:407] Connection header received
0926 13:44:55.925833 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:55.925836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:55.925873 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:55.925927 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:55.926588 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.927062 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.927903 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.928127 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.928197 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:55.930323 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:55.930326 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:55.930328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:55.930348 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:55.931535 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:55.931922 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:55.931925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:55.931926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:55.931993 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:55.932191 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:55.932193 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:55.932194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:55.932325 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:55.932410 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:55.932644 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 13:44:55.932730 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 13:44:55 dist-test-slave-hc1k krb5kdc[13494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 13:44:55 dist-test-slave-hc1k krb5kdc[13494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.65@KRBTEST.COM
I20260926 13:44:55.954392 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:55.947760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52956 (local address 127.12.170.65:46001)
0926 13:44:55.947875 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:55.947879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:55.948005 (+ 126us) server_negotiation.cc:407] Connection header received
0926 13:44:55.948118 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:55.948121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:55.948158 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:55.948223 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:55.948804 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.949260 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.950013 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:55.950210 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:55.950274 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:55.952241 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:55.952244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:55.952246 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:55.952265 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:55.953258 (+ 993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:55.953575 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:55.953577 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:55.953579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:55.953644 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:55.953816 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:55.953817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:55.953818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:55.953944 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:55.954027 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:55.954207 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 13:44:55.954286 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 13:44:55.955569 13644 tablet_service.cc:2908] Rejecting scan request for tablet 03c154b4fe90438caa33470cbb4e4475: Uninitialized: clean time has not yet been initialized
I20260926 13:44:55.957320 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 13601
I20260926 13:44:55.962100 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 13735
I20260926 13:44:55.966440 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 13869
I20260926 13:44:55.970489 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 13507
2026-09-26T13:44:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2034 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:56 dist-test-slave-hc1k krb5kdc[14039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:56 dist-test-slave-hc1k krb5kdc[14039](info): set up 2 sockets
Sep 26 13:44:56 dist-test-slave-hc1k krb5kdc[14039](info): commencing operation
krb5kdc: starting...
W20260926 13:44:56.907224 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.917s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:44:56 dist-test-slave-hc1k krb5kdc[14039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:44:57.012245 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:38031
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:46335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:38031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:57.115511 14052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:57.115741 14052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:57.115794 14052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:57.119603 14052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:57.119668 14052 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 13:44:57.119693 14052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:57.119721 14052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:57.119750 14052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:57.119772 14052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:57.124665 14052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:46335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:38031
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:38031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:57.125509 14052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:57.126374 14052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:57.133786 14057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:57.133786 14058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:57.133909 14052 server_base.cc:1034] running on GCE node
W20260926 13:44:57.133842 14060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:57.134552 14052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:57.135488 14052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:57.136710 14052 hybrid_clock.cc:648] HybridClock initialized: now 1790430297136682 us; error 35 us; skew 500 ppm
Sep 26 13:44:57 dist-test-slave-hc1k krb5kdc[14039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430297, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:57.142637 14052 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:44:57.143442 14052 webserver.cc:460] Webserver started at http://127.12.170.126:33413/ using document root <none> and password file <none>
I20260926 13:44:57.143988 14052 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:57.144052 14052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:57.144276 14052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:57.146145 14052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "486bfe16915e479ea3d6542a0ab35e49"
format_stamp: "Formatted at 2026-09-26 13:44:57 on dist-test-slave-hc1k"
server_key: "6aab4d627fa96dd205dfef8cd36d3121"
server_key_iv: "7ac254b3bebd8931a9c5b204044c616a"
server_key_version: "encryptionkey@0"
I20260926 13:44:57.146641 14052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "486bfe16915e479ea3d6542a0ab35e49"
format_stamp: "Formatted at 2026-09-26 13:44:57 on dist-test-slave-hc1k"
server_key: "6aab4d627fa96dd205dfef8cd36d3121"
server_key_iv: "7ac254b3bebd8931a9c5b204044c616a"
server_key_version: "encryptionkey@0"
I20260926 13:44:57.149969 14052 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:44:57.152105 14067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:57.153162 14052 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:44:57.153266 14052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "486bfe16915e479ea3d6542a0ab35e49"
format_stamp: "Formatted at 2026-09-26 13:44:57 on dist-test-slave-hc1k"
server_key: "6aab4d627fa96dd205dfef8cd36d3121"
server_key_iv: "7ac254b3bebd8931a9c5b204044c616a"
server_key_version: "encryptionkey@0"
I20260926 13:44:57.153353 14052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:57.175634 14052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:57.178098 14052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:57.178248 14052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:57.185475 14052 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:38031
I20260926 13:44:57.185541 14129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:38031 every 8 connection(s)
I20260926 13:44:57.186329 14052 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:57.186801 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 14052
I20260926 13:44:57.186901 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:44:57.187142 12969 external_mini_cluster.cc:1428] Setting key 40816748558347f82ff5c5a6f9471b0b
I20260926 13:44:57.189100 14130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:57.195119 14130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486bfe16915e479ea3d6542a0ab35e49: Bootstrap starting.
Sep 26 13:44:57 dist-test-slave-hc1k krb5kdc[14039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:44:57.197345 14130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 486bfe16915e479ea3d6542a0ab35e49: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:57.198338 14130 log.cc:824] T 00000000000000000000000000000000 P 486bfe16915e479ea3d6542a0ab35e49: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:57.200130 14130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486bfe16915e479ea3d6542a0ab35e49: No bootstrap required, opened a new log
I20260926 13:44:57.200424 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:57.188417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38644 (local address 127.12.170.126:38031)
0926 13:44:57.188793 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:57.188819 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:57.188853 (+ 34us) server_negotiation.cc:407] Connection header received
0926 13:44:57.189507 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:57.189523 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:57.189820 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:57.190083 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:57.190906 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.191937 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.193069 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.193505 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.193618 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:57.196222 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:57.196227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:57.196238 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:57.196263 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:57.197773 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:57.198174 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:57.198179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:57.198183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:57.198283 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:57.198525 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:57.198529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:57.198530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:57.198986 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:57.199170 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:57.199432 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 13:44:57.199723 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":121,"threads_started":1}
I20260926 13:44:57.203038 14130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 486bfe16915e479ea3d6542a0ab35e49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486bfe16915e479ea3d6542a0ab35e49" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 38031 } }
I20260926 13:44:57.203253 14130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 486bfe16915e479ea3d6542a0ab35e49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:57.203323 14130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 486bfe16915e479ea3d6542a0ab35e49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486bfe16915e479ea3d6542a0ab35e49, State: Initialized, Role: FOLLOWER
I20260926 13:44:57.203771 14130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 486bfe16915e479ea3d6542a0ab35e49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486bfe16915e479ea3d6542a0ab35e49" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 38031 } }
I20260926 13:44:57.203899 14130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 486bfe16915e479ea3d6542a0ab35e49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:57.203961 14130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 486bfe16915e479ea3d6542a0ab35e49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:57.204051 14130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 486bfe16915e479ea3d6542a0ab35e49 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:57.204860 14130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 486bfe16915e479ea3d6542a0ab35e49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486bfe16915e479ea3d6542a0ab35e49" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 38031 } }
I20260926 13:44:57.205159 14130 leader_election.cc:302] T 00000000000000000000000000000000 P 486bfe16915e479ea3d6542a0ab35e49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 486bfe16915e479ea3d6542a0ab35e49; no voters:
I20260926 13:44:57.205407 14130 leader_election.cc:288] T 00000000000000000000000000000000 P 486bfe16915e479ea3d6542a0ab35e49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:57.205515 14135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 486bfe16915e479ea3d6542a0ab35e49 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:57.205735 14135 raft_consensus.cc:686] T 00000000000000000000000000000000 P 486bfe16915e479ea3d6542a0ab35e49 [term 1 LEADER]: Becoming Leader. State: Replica: 486bfe16915e479ea3d6542a0ab35e49, State: Running, Role: LEADER
I20260926 13:44:57.206044 14135 consensus_queue.cc:237] T 00000000000000000000000000000000 P 486bfe16915e479ea3d6542a0ab35e49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486bfe16915e479ea3d6542a0ab35e49" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 38031 } }
I20260926 13:44:57.206569 14130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 486bfe16915e479ea3d6542a0ab35e49 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:57.207098 14137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486bfe16915e479ea3d6542a0ab35e49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 486bfe16915e479ea3d6542a0ab35e49. Latest consensus state: current_term: 1 leader_uuid: "486bfe16915e479ea3d6542a0ab35e49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486bfe16915e479ea3d6542a0ab35e49" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 38031 } } }
I20260926 13:44:57.207217 14137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486bfe16915e479ea3d6542a0ab35e49 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:57.207542 14141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:57.207954 14136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486bfe16915e479ea3d6542a0ab35e49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "486bfe16915e479ea3d6542a0ab35e49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486bfe16915e479ea3d6542a0ab35e49" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 38031 } } }
I20260926 13:44:57.208133 14136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486bfe16915e479ea3d6542a0ab35e49 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:57.210107 14141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:57.214715 14141 catalog_manager.cc:1347] Generated new cluster ID: 7b4491f1463945e5b69c3675d89a56e8
I20260926 13:44:57.214767 14141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:57.237445 14141 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:57.238660 14141 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:57.244027 14141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 486bfe16915e479ea3d6542a0ab35e49: Generated new TSK 0
I20260926 13:44:57.244779 14141 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.170.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
I20260926 13:44:57.271961 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:0
--local_ip_for_outbound_sockets=127.12.170.65
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:38031
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:46335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:57.374589 14156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:57.374814 14156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:57.374871 14156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:57.378702 14156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:57.378751 14156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:57.378801 14156 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:44:57.378854 14156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:44:57.383778 14156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:46335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.170.126:38031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:57.384648 14156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:57.385532 14156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:57.393931 14161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:57.393908 14164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:57.393956 14162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:57.394209 14156 server_base.cc:1034] running on GCE node
I20260926 13:44:57.394546 14156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:57.395087 14156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:57.396226 14156 hybrid_clock.cc:648] HybridClock initialized: now 1790430297396211 us; error 27 us; skew 500 ppm
Sep 26 13:44:57 dist-test-slave-hc1k krb5kdc[14039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430297, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:57.398679 14156 init.cc:377] Logged in from keytab as kudu/127.12.170.65@KRBTEST.COM (short username kudu)
I20260926 13:44:57.399444 14156 webserver.cc:460] Webserver started at http://127.12.170.65:45005/ using document root <none> and password file <none>
I20260926 13:44:57.399955 14156 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:57.400025 14156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:57.400211 14156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:57.402072 14156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-0/data/instance:
uuid: "ea8b31356cc244ee960bc10878467e8a"
format_stamp: "Formatted at 2026-09-26 13:44:57 on dist-test-slave-hc1k"
server_key: "ceb6d183f70ee2b80399edd9714aa801"
server_key_iv: "6de03ba4e24e9f4d0a884a7958f794af"
server_key_version: "encryptionkey@0"
I20260926 13:44:57.402568 14156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance:
uuid: "ea8b31356cc244ee960bc10878467e8a"
format_stamp: "Formatted at 2026-09-26 13:44:57 on dist-test-slave-hc1k"
server_key: "ceb6d183f70ee2b80399edd9714aa801"
server_key_iv: "6de03ba4e24e9f4d0a884a7958f794af"
server_key_version: "encryptionkey@0"
I20260926 13:44:57.405887 14156 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:44:57.408197 14171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:57.409255 14156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:44:57.409350 14156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "ea8b31356cc244ee960bc10878467e8a"
format_stamp: "Formatted at 2026-09-26 13:44:57 on dist-test-slave-hc1k"
server_key: "ceb6d183f70ee2b80399edd9714aa801"
server_key_iv: "6de03ba4e24e9f4d0a884a7958f794af"
server_key_version: "encryptionkey@0"
I20260926 13:44:57.409433 14156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:57.434684 14156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:57.437439 14156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:57.437594 14156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:57.438833 14156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:57.438880 14156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:57.438930 14156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:57.438958 14156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 13:44:57 dist-test-slave-hc1k krb5kdc[14039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430297, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:44:57.448437 14156 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:36001
I20260926 13:44:57.448489 14286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:36001 every 8 connection(s)
I20260926 13:44:57.449388 14156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:44:57.451404 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:57.439558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:50745 (local address 127.12.170.126:38031)
0926 13:44:57.439689 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:57.439694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:57.440637 (+ 943us) server_negotiation.cc:407] Connection header received
0926 13:44:57.441500 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:57.441504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:57.441557 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:57.441623 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:57.443113 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.443930 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.445201 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.445436 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.445523 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:57.448153 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:57.448157 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:57.448159 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:57.448186 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:57.449650 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:57.450166 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:57.450170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:57.450172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:57.450259 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:57.450526 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:57.450528 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:57.450529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:57.450685 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:57.450753 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:57.451155 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 13:44:57.451258 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 13:44:57.452049 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:57.440062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:38031 (local address 127.12.170.65:50745)
0926 13:44:57.440490 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:57.440527 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:57.441276 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:57.442001 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:57.442007 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:57.442419 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:57.442951 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:57.442970 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.444042 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.444045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:57.445094 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:57.445103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.445583 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.445588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:57.445726 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:57.446675 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:57.446690 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:57.447981 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:57.449756 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:57.449761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:57.449775 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:57.450062 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:57.450336 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:57.450338 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:57.450339 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:57.450453 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:57.450752 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:57.450756 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:57.450960 (+ 204us) client_negotiation.cc:769] Sending connection context
0926 13:44:57.451098 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 13:44:57.451301 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":123,"threads_started":1}
I20260926 13:44:57.453127 14287 heartbeater.cc:347] Connected to a master server at 127.12.170.126:38031
I20260926 13:44:57.453250 14287 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:57.453729 14287 heartbeater.cc:511] Master 127.12.170.126:38031 requested a full tablet report, sending...
I20260926 13:44:57.455127 14084 ts_manager.cc:194] Registered new tserver with Master: ea8b31356cc244ee960bc10878467e8a (127.12.170.65:36001)
I20260926 13:44:57.456578 14084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.65@KRBTEST.COM'} at 127.12.170.65:50745
I20260926 13:44:57.457381 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 14156
I20260926 13:44:57.457481 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance
I20260926 13:44:57.457746 12969 external_mini_cluster.cc:1428] Setting key e49cfba9dd24c89229b3c7f35b60822b
I20260926 13:44:57.461968 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:57.455150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:54335 (local address 127.12.170.126:38031)
0926 13:44:57.455323 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:57.455327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:57.455341 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:44:57.455432 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:57.455435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:57.455471 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:57.455549 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:57.455944 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.456700 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.457671 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.457984 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.458080 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:57.459216 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:57.459220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:57.459222 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:57.459246 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:57.460640 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:57.461052 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:57.461054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:57.461055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:57.461116 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:57.461352 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:57.461354 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:57.461355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:57.461485 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:57.461549 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:57.461787 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 13:44:57.461865 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":111}
I20260926 13:44:57.462042 14293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:57.455078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:38031 (local address 127.12.170.65:54335)
0926 13:44:57.455291 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:57.455302 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:57.455363 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:57.455580 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:57.455582 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:57.455761 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:57.455845 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:57.455850 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.456810 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.456812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:57.457554 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:57.457575 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.458184 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.458187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:57.458272 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:57.458804 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:57.458814 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:57.459097 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:57.460735 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:57.460738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:57.460741 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:57.460970 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:57.461185 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:57.461188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:57.461190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:57.461281 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:57.461563 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:57.461566 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:57.461645 (+ 79us) client_negotiation.cc:769] Sending connection context
0926 13:44:57.461756 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 13:44:57.461869 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for kudu/127.12.170.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
I20260926 13:44:57.480551 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:0
--local_ip_for_outbound_sockets=127.12.170.66
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:38031
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:46335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:57.581389 14296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:57.581610 14296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:57.581663 14296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:57.585527 14296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:57.585578 14296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:57.585628 14296 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:44:57.585676 14296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:44:57.590579 14296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:46335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.170.126:38031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:57.591491 14296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:57.592425 14296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:57.601265 14304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:57.601359 14296 server_base.cc:1034] running on GCE node
W20260926 13:44:57.601249 14301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:57.601249 14302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:57.601910 14296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:57.602540 14296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:57.603726 14296 hybrid_clock.cc:648] HybridClock initialized: now 1790430297603706 us; error 35 us; skew 500 ppm
Sep 26 13:44:57 dist-test-slave-hc1k krb5kdc[14039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430297, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:57.606218 14296 init.cc:377] Logged in from keytab as kudu/127.12.170.66@KRBTEST.COM (short username kudu)
I20260926 13:44:57.607036 14296 webserver.cc:460] Webserver started at http://127.12.170.66:46711/ using document root <none> and password file <none>
I20260926 13:44:57.607564 14296 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:57.607626 14296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:57.607846 14296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:57.609786 14296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-1/data/instance:
uuid: "52709b0b591d404c9428fcb5155133e8"
format_stamp: "Formatted at 2026-09-26 13:44:57 on dist-test-slave-hc1k"
server_key: "46f4cd8154080fcb0685d3cc921a7ad0"
server_key_iv: "1cbbb4a1e7c70932b12b9e44f0b42b93"
server_key_version: "encryptionkey@0"
I20260926 13:44:57.610282 14296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance:
uuid: "52709b0b591d404c9428fcb5155133e8"
format_stamp: "Formatted at 2026-09-26 13:44:57 on dist-test-slave-hc1k"
server_key: "46f4cd8154080fcb0685d3cc921a7ad0"
server_key_iv: "1cbbb4a1e7c70932b12b9e44f0b42b93"
server_key_version: "encryptionkey@0"
I20260926 13:44:57.613667 14296 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:44:57.615815 14311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:57.616909 14296 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:44:57.617005 14296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "52709b0b591d404c9428fcb5155133e8"
format_stamp: "Formatted at 2026-09-26 13:44:57 on dist-test-slave-hc1k"
server_key: "46f4cd8154080fcb0685d3cc921a7ad0"
server_key_iv: "1cbbb4a1e7c70932b12b9e44f0b42b93"
server_key_version: "encryptionkey@0"
I20260926 13:44:57.617087 14296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:57.630111 14296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:57.632850 14296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:57.633003 14296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:57.634264 14296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:57.634310 14296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:57.634358 14296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:57.634387 14296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 13:44:57 dist-test-slave-hc1k krb5kdc[14039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430297, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:44:57.644068 14296 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:43895
I20260926 13:44:57.644126 14426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:43895 every 8 connection(s)
I20260926 13:44:57.645072 14296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:44:57.646790 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:57.634874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:39265 (local address 127.12.170.126:38031)
0926 13:44:57.635016 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:57.635022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:57.636273 (+ 1251us) server_negotiation.cc:407] Connection header received
0926 13:44:57.637479 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:57.637482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:57.637527 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:57.637603 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:57.639057 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.639736 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.640865 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.641091 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.641179 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:57.643459 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:57.643463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:57.643466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:57.643493 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:57.645070 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:57.645551 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:57.645554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:57.645557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:57.645645 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:57.645936 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:57.645939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:57.645940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:57.646115 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:57.646195 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:57.646563 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 13:44:57.646653 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 13:44:57.647511 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:57.635673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:38031 (local address 127.12.170.66:39265)
0926 13:44:57.636118 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:57.636169 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:57.637247 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:57.637783 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:57.637792 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:57.638243 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:57.638883 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:57.638904 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.639852 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.639856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:57.640751 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:57.640761 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.641220 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.641224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:57.641371 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:57.641880 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:57.641894 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:57.643299 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:57.645195 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:57.645201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:57.645213 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:57.645439 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:57.645740 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:57.645743 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:57.645745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:57.645862 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:57.646193 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:57.646197 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:57.646391 (+ 194us) client_negotiation.cc:769] Sending connection context
0926 13:44:57.646526 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 13:44:57.646741 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":97,"threads_started":1}
I20260926 13:44:57.648586 14427 heartbeater.cc:347] Connected to a master server at 127.12.170.126:38031
I20260926 13:44:57.648710 14427 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:57.649204 14427 heartbeater.cc:511] Master 127.12.170.126:38031 requested a full tablet report, sending...
I20260926 13:44:57.650175 14083 ts_manager.cc:194] Registered new tserver with Master: 52709b0b591d404c9428fcb5155133e8 (127.12.170.66:43895)
I20260926 13:44:57.650844 14083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.66@KRBTEST.COM'} at 127.12.170.66:39265
I20260926 13:44:57.655093 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 14296
I20260926 13:44:57.655194 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance
I20260926 13:44:57.655423 12969 external_mini_cluster.cc:1428] Setting key 6cdee7ab7e2225e12caff9e6b83050fa
I20260926 13:44:57.656823 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:57.650663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:45911 (local address 127.12.170.126:38031)
0926 13:44:57.650774 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:57.650778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:57.650958 (+ 180us) server_negotiation.cc:407] Connection header received
0926 13:44:57.651110 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:57.651113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:57.651152 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:57.651223 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:57.651574 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.652115 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.652868 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.653081 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.653159 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:57.653968 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:57.653971 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:57.653974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:57.653996 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:57.655317 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:57.655735 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:57.655739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:57.655741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:57.655813 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:57.656073 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:57.656076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:57.656077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:57.656264 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:57.656344 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:57.656610 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 13:44:57.656699 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 13:44:57.656836 14433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:57.650608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:38031 (local address 127.12.170.66:45911)
0926 13:44:57.650895 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:57.650906 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:57.651014 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:57.651254 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:57.651257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:57.651393 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:57.651478 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:57.651483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.652233 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.652236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:57.652772 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:57.652779 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.653169 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.653173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:57.653266 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:57.653592 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:57.653602 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:57.653856 (+ 254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:57.655406 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:57.655409 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:57.655411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:57.655626 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:57.655887 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:57.655890 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:57.655892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:57.655987 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:57.656350 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:57.656353 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:57.656433 (+ 80us) client_negotiation.cc:769] Sending connection context
0926 13:44:57.656551 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 13:44:57.656673 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for kudu/127.12.170.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
I20260926 13:44:57.677598 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:0
--local_ip_for_outbound_sockets=127.12.170.67
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:38031
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:46335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:57.779417 14436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:57.779635 14436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:57.779721 14436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:57.783627 14436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:57.783689 14436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:57.783742 14436 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:44:57.783797 14436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:44:57.788671 14436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:46335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.170.126:38031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:57.789565 14436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:57.790455 14436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:44:57.799211 14436 server_base.cc:1034] running on GCE node
W20260926 13:44:57.799180 14444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:57.799178 14441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:57.799185 14442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:57.799755 14436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:57.800319 14436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:57.801460 14436 hybrid_clock.cc:648] HybridClock initialized: now 1790430297801445 us; error 32 us; skew 500 ppm
Sep 26 13:44:57 dist-test-slave-hc1k krb5kdc[14039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430297, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:57.803951 14436 init.cc:377] Logged in from keytab as kudu/127.12.170.67@KRBTEST.COM (short username kudu)
I20260926 13:44:57.804680 14436 webserver.cc:460] Webserver started at http://127.12.170.67:44529/ using document root <none> and password file <none>
I20260926 13:44:57.805145 14436 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:57.805197 14436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:57.805344 14436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:57.807250 14436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-2/data/instance:
uuid: "99f89e8e3392413ab58b7363975ea0ea"
format_stamp: "Formatted at 2026-09-26 13:44:57 on dist-test-slave-hc1k"
server_key: "2b6050af6f7d4e3dd6935323d84ada6b"
server_key_iv: "f20b7403e7b722956dffa0b8a5921d78"
server_key_version: "encryptionkey@0"
I20260926 13:44:57.807798 14436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance:
uuid: "99f89e8e3392413ab58b7363975ea0ea"
format_stamp: "Formatted at 2026-09-26 13:44:57 on dist-test-slave-hc1k"
server_key: "2b6050af6f7d4e3dd6935323d84ada6b"
server_key_iv: "f20b7403e7b722956dffa0b8a5921d78"
server_key_version: "encryptionkey@0"
I20260926 13:44:57.810998 14436 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:44:57.813203 14451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:57.814311 14436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:44:57.814395 14436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "99f89e8e3392413ab58b7363975ea0ea"
format_stamp: "Formatted at 2026-09-26 13:44:57 on dist-test-slave-hc1k"
server_key: "2b6050af6f7d4e3dd6935323d84ada6b"
server_key_iv: "f20b7403e7b722956dffa0b8a5921d78"
server_key_version: "encryptionkey@0"
I20260926 13:44:57.814465 14436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:57.836259 14436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:57.838940 14436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:57.839075 14436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:57.840339 14436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:57.840384 14436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:57.840420 14436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:57.840433 14436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 13:44:57 dist-test-slave-hc1k krb5kdc[14039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430297, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:44:57.849959 14436 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:45747
I20260926 13:44:57.850069 14566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:45747 every 8 connection(s)
I20260926 13:44:57.850792 14436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:44:57.852334 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 14436
I20260926 13:44:57.852453 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance
I20260926 13:44:57.852797 12969 external_mini_cluster.cc:1428] Setting key 014a7a8545576417fcb97909f260f041
I20260926 13:44:57.852895 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:57.840996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:39665 (local address 127.12.170.126:38031)
0926 13:44:57.841109 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:57.841113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:57.842108 (+ 995us) server_negotiation.cc:407] Connection header received
0926 13:44:57.843094 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:57.843098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:57.843161 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:57.843220 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:57.844550 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.845132 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.846315 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.846617 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.846724 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:57.849329 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:57.849334 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:57.849336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:57.849367 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:57.850857 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:57.851408 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:57.851411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:57.851413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:57.851502 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:57.851861 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:57.851863 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:57.851864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:57.852029 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:57.852104 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:57.852654 (+ 550us) server_negotiation.cc:299] Negotiation successful
0926 13:44:57.852762 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 13:44:57.853737 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:57.841441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:38031 (local address 127.12.170.67:39665)
0926 13:44:57.841912 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:57.841952 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:57.842819 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:57.843584 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:57.843593 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:57.843954 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:57.844384 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:57.844397 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.845261 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.845265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:57.846181 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:57.846191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.846772 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.846776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:57.846969 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:57.847486 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:57.847505 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:57.849162 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:57.850965 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:57.850970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:57.850983 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:57.851288 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:57.851594 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:57.851596 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:57.851598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:57.851773 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:57.852115 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:57.852121 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:57.852395 (+ 274us) client_negotiation.cc:769] Sending connection context
0926 13:44:57.852592 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 13:44:57.852881 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":133,"threads_started":1}
I20260926 13:44:57.854799 14567 heartbeater.cc:347] Connected to a master server at 127.12.170.126:38031
I20260926 13:44:57.854930 14567 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:57.855472 14567 heartbeater.cc:511] Master 127.12.170.126:38031 requested a full tablet report, sending...
I20260926 13:44:57.856498 14083 ts_manager.cc:194] Registered new tserver with Master: 99f89e8e3392413ab58b7363975ea0ea (127.12.170.67:45747)
I20260926 13:44:57.857221 14083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.67@KRBTEST.COM'} at 127.12.170.67:39665
I20260926 13:44:57.862759 14573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:57.856876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:38031 (local address 127.12.170.67:40911)
0926 13:44:57.857144 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:57.857159 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:57.857256 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:57.857496 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:57.857504 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:57.857657 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:57.857747 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:57.857753 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.858436 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.858439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:57.859201 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:57.859211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.859555 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.859558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:57.859623 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:57.859989 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:57.860002 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:57.860265 (+ 263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:57.861692 (+ 1427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:57.861695 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:57.861696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:57.861885 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:57.862095 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:57.862097 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:57.862098 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:57.862162 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:57.862415 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:57.862416 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:57.862480 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 13:44:57.862558 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 13:44:57.862646 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":94,"threads_started":1}
I20260926 13:44:57.862776 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:57.856937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:40911 (local address 127.12.170.126:38031)
0926 13:44:57.857043 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:57.857047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:57.857225 (+ 178us) server_negotiation.cc:407] Connection header received
0926 13:44:57.857344 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:57.857347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:57.857388 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:57.857455 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:57.857853 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.858320 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.859291 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.859480 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.859541 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:57.860379 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:57.860381 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:57.860383 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:57.860402 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:57.861589 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:57.861964 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:57.861966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:57.861967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:57.862029 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:57.862220 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:57.862222 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:57.862223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:57.862356 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:57.862420 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:57.862607 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 13:44:57.862681 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 13:44:57.866374 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 13:44:57 dist-test-slave-hc1k krb5kdc[14039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430297, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 13:44:57 dist-test-slave-hc1k krb5kdc[14039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430297, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:44:57.888012 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:57.881395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38658 (local address 127.12.170.126:38031)
0926 13:44:57.881494 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:57.881497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:57.881507 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:44:57.881547 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:57.881549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:57.881580 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:57.881644 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:57.882214 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.882706 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.883417 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.883617 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.883723 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:57.885740 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:57.885742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:57.885744 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:57.885763 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:57.886834 (+ 1071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:57.887159 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:57.887161 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:57.887162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:57.887219 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:57.887396 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:57.887398 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:57.887399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:57.887546 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:57.887635 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:57.887826 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 13:44:57.887905 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 13:44:57.890604 14083 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:44:57.893438 14083 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:44:57.906499 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:57.900728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:43895 (local address 127.0.0.1:59502)
0926 13:44:57.901333 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:57.901379 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:57.901505 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:57.902700 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:57.902703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:57.902719 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:57.902866 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:57.902872 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.903798 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.903801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:57.904983 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:57.904993 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.905972 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.905976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:57.906092 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:57.906133 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 13:44:57.906223 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 13:44:57.906351 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":23,"spinlock_wait_cycles":2304,"thread_start_us":69,"threads_started":1}
I20260926 13:44:57.906644 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:57.901025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:36001 (local address 127.0.0.1:38230)
0926 13:44:57.901967 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:57.901979 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:57.902074 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:57.902588 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:57.902595 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:57.902626 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:57.902856 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:57.902864 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.904272 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.904276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:57.905265 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:57.905272 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.906218 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.906222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:57.906306 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:57.906327 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 13:44:57.906384 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 13:44:57.906450 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"mutex_wait_us":80,"spinlock_wait_cycles":6528,"thread_start_us":71,"threads_started":1}
I20260926 13:44:57.906804 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:57.901490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.12.170.65:36001)
0926 13:44:57.901861 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:57.901869 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:57.902073 (+ 204us) server_negotiation.cc:407] Connection header received
0926 13:44:57.902167 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:57.902171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:57.902310 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:57.902439 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:57.902987 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.904163 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.905362 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.906117 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.906208 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:57.906355 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:57.906408 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 13:44:57.906487 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":100,"threads_started":1}
I20260926 13:44:57.907069 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:57.900874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59502 (local address 127.12.170.66:43895)
0926 13:44:57.902315 (+ 1441us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:57.902327 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:57.902348 (+ 21us) server_negotiation.cc:407] Connection header received
0926 13:44:57.902420 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:57.902425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:57.902582 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:57.902707 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:57.902958 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.903703 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.905114 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.905855 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.905963 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:57.906132 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:57.906839 (+ 707us) server_negotiation.cc:299] Negotiation successful
0926 13:44:57.906933 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1342,"thread_start_us":111,"threads_started":1}
I20260926 13:44:57.909042 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:57.901499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48894 (local address 127.12.170.67:45747)
0926 13:44:57.901981 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:57.901990 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:57.902010 (+ 20us) server_negotiation.cc:407] Connection header received
0926 13:44:57.902086 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:57.902092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:57.902292 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:57.902439 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:57.902987 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.904164 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.906270 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.908038 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.908134 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:57.908275 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:57.908814 (+ 539us) server_negotiation.cc:299] Negotiation successful
0926 13:44:57.908902 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":106,"threads_started":1}
I20260926 13:44:57.908704 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:57.900929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:45747 (local address 127.0.0.1:48894)
0926 13:44:57.901372 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:57.901385 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:57.901505 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:57.902588 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:57.902594 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:57.902625 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:57.902875 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:57.902881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.905354 (+ 2473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.905356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:57.906146 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:57.906153 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:57.908333 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:57.908337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:57.908424 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:57.908445 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 13:44:57.908507 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 13:44:57.908572 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":67,"spinlock_wait_cycles":17536,"thread_start_us":87,"threads_started":1}
I20260926 13:44:57.909406 14361 tablet_service.cc:1467] Processing CreateTablet for tablet 399736efc31546a380dabacf5cc2c7fd (DEFAULT_TABLE table=test-table [id=e40b33010c27443dbe06aab2b8b964c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:57.909567 14221 tablet_service.cc:1467] Processing CreateTablet for tablet 399736efc31546a380dabacf5cc2c7fd (DEFAULT_TABLE table=test-table [id=e40b33010c27443dbe06aab2b8b964c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:57.910383 14361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 399736efc31546a380dabacf5cc2c7fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:57.910492 14221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 399736efc31546a380dabacf5cc2c7fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:57.911577 14501 tablet_service.cc:1467] Processing CreateTablet for tablet 399736efc31546a380dabacf5cc2c7fd (DEFAULT_TABLE table=test-table [id=e40b33010c27443dbe06aab2b8b964c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:57.912518 14501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 399736efc31546a380dabacf5cc2c7fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:57.916747 14590 tablet_bootstrap.cc:492] T 399736efc31546a380dabacf5cc2c7fd P ea8b31356cc244ee960bc10878467e8a: Bootstrap starting.
I20260926 13:44:57.918586 14591 tablet_bootstrap.cc:492] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8: Bootstrap starting.
I20260926 13:44:57.919323 14590 tablet_bootstrap.cc:654] T 399736efc31546a380dabacf5cc2c7fd P ea8b31356cc244ee960bc10878467e8a: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:57.919943 14592 tablet_bootstrap.cc:492] T 399736efc31546a380dabacf5cc2c7fd P 99f89e8e3392413ab58b7363975ea0ea: Bootstrap starting.
I20260926 13:44:57.920193 14590 log.cc:824] T 399736efc31546a380dabacf5cc2c7fd P ea8b31356cc244ee960bc10878467e8a: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:57.920562 14591 tablet_bootstrap.cc:654] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:57.921269 14591 log.cc:824] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:57.921917 14592 tablet_bootstrap.cc:654] T 399736efc31546a380dabacf5cc2c7fd P 99f89e8e3392413ab58b7363975ea0ea: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:57.922289 14590 tablet_bootstrap.cc:492] T 399736efc31546a380dabacf5cc2c7fd P ea8b31356cc244ee960bc10878467e8a: No bootstrap required, opened a new log
I20260926 13:44:57.922501 14590 ts_tablet_manager.cc:1397] T 399736efc31546a380dabacf5cc2c7fd P ea8b31356cc244ee960bc10878467e8a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 13:44:57.922672 14592 log.cc:824] T 399736efc31546a380dabacf5cc2c7fd P 99f89e8e3392413ab58b7363975ea0ea: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:57.922741 14591 tablet_bootstrap.cc:492] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8: No bootstrap required, opened a new log
I20260926 13:44:57.922864 14591 ts_tablet_manager.cc:1397] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 13:44:57.924655 14592 tablet_bootstrap.cc:492] T 399736efc31546a380dabacf5cc2c7fd P 99f89e8e3392413ab58b7363975ea0ea: No bootstrap required, opened a new log
I20260926 13:44:57.924823 14592 ts_tablet_manager.cc:1397] T 399736efc31546a380dabacf5cc2c7fd P 99f89e8e3392413ab58b7363975ea0ea: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 13:44:57.925820 14591 raft_consensus.cc:348] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } } peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } }
I20260926 13:44:57.926014 14591 raft_consensus.cc:374] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:57.926052 14591 raft_consensus.cc:729] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52709b0b591d404c9428fcb5155133e8, State: Initialized, Role: FOLLOWER
I20260926 13:44:57.926388 14591 consensus_queue.cc:260] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } } peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } }
I20260926 13:44:57.926904 14427 heartbeater.cc:503] Master 127.12.170.126:38031 was elected leader, sending a full tablet report...
I20260926 13:44:57.926923 14590 raft_consensus.cc:348] T 399736efc31546a380dabacf5cc2c7fd P ea8b31356cc244ee960bc10878467e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } } peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } }
I20260926 13:44:57.927261 14590 raft_consensus.cc:374] T 399736efc31546a380dabacf5cc2c7fd P ea8b31356cc244ee960bc10878467e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:57.927356 14590 raft_consensus.cc:729] T 399736efc31546a380dabacf5cc2c7fd P ea8b31356cc244ee960bc10878467e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea8b31356cc244ee960bc10878467e8a, State: Initialized, Role: FOLLOWER
I20260926 13:44:57.927835 14590 consensus_queue.cc:260] T 399736efc31546a380dabacf5cc2c7fd P ea8b31356cc244ee960bc10878467e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } } peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } }
I20260926 13:44:57.928053 14591 ts_tablet_manager.cc:1428] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 13:44:57.928293 14592 raft_consensus.cc:348] T 399736efc31546a380dabacf5cc2c7fd P 99f89e8e3392413ab58b7363975ea0ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } } peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } }
I20260926 13:44:57.928507 14592 raft_consensus.cc:374] T 399736efc31546a380dabacf5cc2c7fd P 99f89e8e3392413ab58b7363975ea0ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:57.928546 14287 heartbeater.cc:503] Master 127.12.170.126:38031 was elected leader, sending a full tablet report...
I20260926 13:44:57.928570 14592 raft_consensus.cc:729] T 399736efc31546a380dabacf5cc2c7fd P 99f89e8e3392413ab58b7363975ea0ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f89e8e3392413ab58b7363975ea0ea, State: Initialized, Role: FOLLOWER
I20260926 13:44:57.928969 14590 ts_tablet_manager.cc:1428] T 399736efc31546a380dabacf5cc2c7fd P ea8b31356cc244ee960bc10878467e8a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 13:44:57.928966 14592 consensus_queue.cc:260] T 399736efc31546a380dabacf5cc2c7fd P 99f89e8e3392413ab58b7363975ea0ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } } peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } }
I20260926 13:44:57.929672 14567 heartbeater.cc:503] Master 127.12.170.126:38031 was elected leader, sending a full tablet report...
I20260926 13:44:57.929843 14592 ts_tablet_manager.cc:1428] T 399736efc31546a380dabacf5cc2c7fd P 99f89e8e3392413ab58b7363975ea0ea: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260926 13:44:57.950402 14288 tablet.cc:2369] T 399736efc31546a380dabacf5cc2c7fd P ea8b31356cc244ee960bc10878467e8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:44:58.010718 14596 raft_consensus.cc:482] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:44:58.010898 14596 raft_consensus.cc:504] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } } peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } }
I20260926 13:44:58.012100 14596 leader_election.cc:288] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99f89e8e3392413ab58b7363975ea0ea (127.12.170.67:45747), ea8b31356cc244ee960bc10878467e8a (127.12.170.65:36001)
I20260926 13:44:58.015715 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:58.012292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:45747 (local address 127.12.170.66:53149)
0926 13:44:58.012475 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:58.012490 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:58.012594 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:58.012839 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:58.012842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:58.012860 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:58.012959 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:58.012964 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.013735 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.013738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:58.014722 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:58.014732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.015352 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.015355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:58.015458 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:58.015477 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 13:44:58.015531 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 13:44:58.015592 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260926 13:44:58.015795 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:58.012456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:53149 (local address 127.12.170.67:45747)
0926 13:44:58.012575 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:58.012579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:58.012595 (+ 16us) server_negotiation.cc:407] Connection header received
0926 13:44:58.012699 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:58.012702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:58.012741 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:58.012809 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:58.013068 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.013636 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.014841 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.015256 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.015367 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:58.015417 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:58.015579 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 13:44:58.015669 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:44:58.016381 14521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "399736efc31546a380dabacf5cc2c7fd" candidate_uuid: "52709b0b591d404c9428fcb5155133e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f89e8e3392413ab58b7363975ea0ea" is_pre_election: true
I20260926 13:44:58.016460 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:58.012433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:53025 (local address 127.12.170.65:36001)
0926 13:44:58.012543 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:58.012547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:58.012762 (+ 215us) server_negotiation.cc:407] Connection header received
0926 13:44:58.012906 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:58.012909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:58.012950 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:58.013008 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:58.013620 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.014159 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.015285 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.015843 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.015956 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:58.015999 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:58.016316 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 13:44:58.016377 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 13:44:58.016479 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:58.012359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:36001 (local address 127.12.170.66:53025)
0926 13:44:58.012685 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:58.012697 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:58.012777 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:58.013026 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:58.013029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:58.013043 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:58.013487 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:58.013494 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.014267 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.014270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:58.015168 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:58.015176 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.016085 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.016090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:58.016182 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:58.016210 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 13:44:58.016281 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 13:44:58.016340 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":1,"thread_start_us":135,"threads_started":1}
I20260926 13:44:58.016702 14521 raft_consensus.cc:2454] T 399736efc31546a380dabacf5cc2c7fd P 99f89e8e3392413ab58b7363975ea0ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52709b0b591d404c9428fcb5155133e8 in term 0.
I20260926 13:44:58.017144 14315 leader_election.cc:302] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52709b0b591d404c9428fcb5155133e8, 99f89e8e3392413ab58b7363975ea0ea; no voters:
I20260926 13:44:58.017134 14241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "399736efc31546a380dabacf5cc2c7fd" candidate_uuid: "52709b0b591d404c9428fcb5155133e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea8b31356cc244ee960bc10878467e8a" is_pre_election: true
I20260926 13:44:58.017374 14241 raft_consensus.cc:2454] T 399736efc31546a380dabacf5cc2c7fd P ea8b31356cc244ee960bc10878467e8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52709b0b591d404c9428fcb5155133e8 in term 0.
I20260926 13:44:58.017383 14596 raft_consensus.cc:2781] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:44:58.017450 14596 raft_consensus.cc:482] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:44:58.017479 14596 raft_consensus.cc:3037] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:58.018476 14596 raft_consensus.cc:504] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } } peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } }
I20260926 13:44:58.018781 14596 leader_election.cc:288] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8 [CANDIDATE]: Term 1 election: Requested vote from peers 99f89e8e3392413ab58b7363975ea0ea (127.12.170.67:45747), ea8b31356cc244ee960bc10878467e8a (127.12.170.65:36001)
I20260926 13:44:58.019065 14241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "399736efc31546a380dabacf5cc2c7fd" candidate_uuid: "52709b0b591d404c9428fcb5155133e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea8b31356cc244ee960bc10878467e8a"
I20260926 13:44:58.019052 14521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "399736efc31546a380dabacf5cc2c7fd" candidate_uuid: "52709b0b591d404c9428fcb5155133e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f89e8e3392413ab58b7363975ea0ea"
I20260926 13:44:58.019161 14241 raft_consensus.cc:3037] T 399736efc31546a380dabacf5cc2c7fd P ea8b31356cc244ee960bc10878467e8a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:58.019169 14521 raft_consensus.cc:3037] T 399736efc31546a380dabacf5cc2c7fd P 99f89e8e3392413ab58b7363975ea0ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:58.020257 14241 raft_consensus.cc:2454] T 399736efc31546a380dabacf5cc2c7fd P ea8b31356cc244ee960bc10878467e8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52709b0b591d404c9428fcb5155133e8 in term 1.
I20260926 13:44:58.020433 14521 raft_consensus.cc:2454] T 399736efc31546a380dabacf5cc2c7fd P 99f89e8e3392413ab58b7363975ea0ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52709b0b591d404c9428fcb5155133e8 in term 1.
I20260926 13:44:58.020531 14314 leader_election.cc:302] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52709b0b591d404c9428fcb5155133e8, ea8b31356cc244ee960bc10878467e8a; no voters:
I20260926 13:44:58.020750 14596 raft_consensus.cc:2781] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:58.020952 14596 raft_consensus.cc:686] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8 [term 1 LEADER]: Becoming Leader. State: Replica: 52709b0b591d404c9428fcb5155133e8, State: Running, Role: LEADER
I20260926 13:44:58.021240 14596 consensus_queue.cc:237] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } } peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } }
I20260926 13:44:58.024160 14082 catalog_manager.cc:5495] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52709b0b591d404c9428fcb5155133e8 (127.12.170.66). New cstate: current_term: 1 leader_uuid: "52709b0b591d404c9428fcb5155133e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } health_report { overall_health: UNKNOWN } } }
I20260926 13:44:58.061671 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:58.058394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59518 (local address 127.12.170.66:43895)
0926 13:44:58.058499 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:58.058502 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:58.058514 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:44:58.058707 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:58.058709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:58.058759 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:58.058840 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:44:58.059018 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.059528 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.060368 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.060592 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.060670 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:58.060948 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:44:58.061328 (+ 380us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:44:58.061417 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:58.061521 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 13:44:58.061569 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 13:44:58.067041 14521 raft_consensus.cc:1261] T 399736efc31546a380dabacf5cc2c7fd P 99f89e8e3392413ab58b7363975ea0ea [term 1 FOLLOWER]: Refusing update from remote peer 52709b0b591d404c9428fcb5155133e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:44:58.067044 14241 raft_consensus.cc:1261] T 399736efc31546a380dabacf5cc2c7fd P ea8b31356cc244ee960bc10878467e8a [term 1 FOLLOWER]: Refusing update from remote peer 52709b0b591d404c9428fcb5155133e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:44:58.067667 14600 consensus_queue.cc:1035] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:44:58.067840 14596 consensus_queue.cc:1035] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:44:58.073180 14603 mvcc.cc:205] Tried to move back new op lower bound from 7333602500876083200 to 7333602500697346048. Current Snapshot: MvccSnapshot[applied={T|T < 7333602500876083200}]
I20260926 13:44:58.084203 14084 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:44:58.084342 14084 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:44:58.086447 14084 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 486bfe16915e479ea3d6542a0ab35e49: Sending DeleteTablet for 3 replicas of tablet 399736efc31546a380dabacf5cc2c7fd
I20260926 13:44:58.087105 14361 tablet_service.cc:1514] Processing DeleteTablet for tablet 399736efc31546a380dabacf5cc2c7fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:44:58 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:59502
I20260926 13:44:58.087106 14501 tablet_service.cc:1514] Processing DeleteTablet for tablet 399736efc31546a380dabacf5cc2c7fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:44:58 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:48894
I20260926 13:44:58.087401 14221 tablet_service.cc:1514] Processing DeleteTablet for tablet 399736efc31546a380dabacf5cc2c7fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:44:58 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:38230
I20260926 13:44:58.087540 14613 tablet_replica.cc:330] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8: stopping tablet replica
I20260926 13:44:58.087808 14615 tablet_replica.cc:330] T 399736efc31546a380dabacf5cc2c7fd P ea8b31356cc244ee960bc10878467e8a: stopping tablet replica
I20260926 13:44:58.087827 14613 raft_consensus.cc:2229] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:44:58.088058 14615 raft_consensus.cc:2229] T 399736efc31546a380dabacf5cc2c7fd P ea8b31356cc244ee960bc10878467e8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:44:58.088127 14613 raft_consensus.cc:2258] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:44:58.088312 14615 raft_consensus.cc:2258] T 399736efc31546a380dabacf5cc2c7fd P ea8b31356cc244ee960bc10878467e8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:44:58.088435 14614 tablet_replica.cc:330] T 399736efc31546a380dabacf5cc2c7fd P 99f89e8e3392413ab58b7363975ea0ea: stopping tablet replica
I20260926 13:44:58.088672 14614 raft_consensus.cc:2229] T 399736efc31546a380dabacf5cc2c7fd P 99f89e8e3392413ab58b7363975ea0ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:44:58.088675 14084 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 13:44:58.088936 14614 raft_consensus.cc:2258] T 399736efc31546a380dabacf5cc2c7fd P 99f89e8e3392413ab58b7363975ea0ea [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 13:44:58.089030 14084 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:44:58.090484 14615 ts_tablet_manager.cc:1905] T 399736efc31546a380dabacf5cc2c7fd P ea8b31356cc244ee960bc10878467e8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:44:58.092293 14613 ts_tablet_manager.cc:1905] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:44:58.093796 14221 tablet_service.cc:1467] Processing CreateTablet for tablet 812fd27453764152a89e22dbc85bdd1f (DEFAULT_TABLE table=test-table [id=8857f8b937d442ed89ea4ae0d98dd6b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:58.093909 14361 tablet_service.cc:1467] Processing CreateTablet for tablet 812fd27453764152a89e22dbc85bdd1f (DEFAULT_TABLE table=test-table [id=8857f8b937d442ed89ea4ae0d98dd6b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:58.094019 14221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 812fd27453764152a89e22dbc85bdd1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:58.094127 14361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 812fd27453764152a89e22dbc85bdd1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:58.094735 14613 ts_tablet_manager.cc:1918] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:44:58.094812 14613 log.cc:1196] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-1/wal/wals/399736efc31546a380dabacf5cc2c7fd
I20260926 13:44:58.095100 14613 ts_tablet_manager.cc:1939] T 399736efc31546a380dabacf5cc2c7fd P 52709b0b591d404c9428fcb5155133e8: Deleting consensus metadata
I20260926 13:44:58.096091 14591 tablet_bootstrap.cc:492] T 812fd27453764152a89e22dbc85bdd1f P 52709b0b591d404c9428fcb5155133e8: Bootstrap starting.
I20260926 13:44:58.096166 14069 catalog_manager.cc:4841] TS 52709b0b591d404c9428fcb5155133e8 (127.12.170.66:43895): tablet 399736efc31546a380dabacf5cc2c7fd (table test-table [id=e40b33010c27443dbe06aab2b8b964c7]) successfully deleted
I20260926 13:44:58.096325 14590 tablet_bootstrap.cc:492] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a: Bootstrap starting.
I20260926 13:44:58.096279 14501 tablet_service.cc:1467] Processing CreateTablet for tablet 812fd27453764152a89e22dbc85bdd1f (DEFAULT_TABLE table=test-table [id=8857f8b937d442ed89ea4ae0d98dd6b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:44:58.096604 14501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 812fd27453764152a89e22dbc85bdd1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:58.096920 14591 tablet_bootstrap.cc:654] T 812fd27453764152a89e22dbc85bdd1f P 52709b0b591d404c9428fcb5155133e8: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:58.097141 14590 tablet_bootstrap.cc:654] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:58.097965 14591 tablet_bootstrap.cc:492] T 812fd27453764152a89e22dbc85bdd1f P 52709b0b591d404c9428fcb5155133e8: No bootstrap required, opened a new log
I20260926 13:44:58.098037 14591 ts_tablet_manager.cc:1397] T 812fd27453764152a89e22dbc85bdd1f P 52709b0b591d404c9428fcb5155133e8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:44:58.098443 14591 raft_consensus.cc:348] T 812fd27453764152a89e22dbc85bdd1f P 52709b0b591d404c9428fcb5155133e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } }
I20260926 13:44:58.098531 14591 raft_consensus.cc:374] T 812fd27453764152a89e22dbc85bdd1f P 52709b0b591d404c9428fcb5155133e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:58.098570 14591 raft_consensus.cc:729] T 812fd27453764152a89e22dbc85bdd1f P 52709b0b591d404c9428fcb5155133e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52709b0b591d404c9428fcb5155133e8, State: Initialized, Role: FOLLOWER
I20260926 13:44:58.098673 14591 consensus_queue.cc:260] T 812fd27453764152a89e22dbc85bdd1f P 52709b0b591d404c9428fcb5155133e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } }
I20260926 13:44:58.098762 14615 ts_tablet_manager.cc:1918] T 399736efc31546a380dabacf5cc2c7fd P ea8b31356cc244ee960bc10878467e8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:44:58.098836 14615 log.cc:1196] T 399736efc31546a380dabacf5cc2c7fd P ea8b31356cc244ee960bc10878467e8a: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-0/wal/wals/399736efc31546a380dabacf5cc2c7fd
I20260926 13:44:58.098886 14591 ts_tablet_manager.cc:1428] T 812fd27453764152a89e22dbc85bdd1f P 52709b0b591d404c9428fcb5155133e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:44:58.098905 14592 tablet_bootstrap.cc:492] T 812fd27453764152a89e22dbc85bdd1f P 99f89e8e3392413ab58b7363975ea0ea: Bootstrap starting.
I20260926 13:44:58.099784 14592 tablet_bootstrap.cc:654] T 812fd27453764152a89e22dbc85bdd1f P 99f89e8e3392413ab58b7363975ea0ea: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:58.090402 14614 ts_tablet_manager.cc:1905] T 399736efc31546a380dabacf5cc2c7fd P 99f89e8e3392413ab58b7363975ea0ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:44:58.100507 14590 tablet_bootstrap.cc:492] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a: No bootstrap required, opened a new log
I20260926 13:44:58.100595 14590 ts_tablet_manager.cc:1397] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 13:44:58.100975 14590 raft_consensus.cc:348] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } }
I20260926 13:44:58.101080 14590 raft_consensus.cc:374] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:58.101116 14590 raft_consensus.cc:729] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea8b31356cc244ee960bc10878467e8a, State: Initialized, Role: FOLLOWER
I20260926 13:44:58.101228 14590 consensus_queue.cc:260] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } }
I20260926 13:44:58.101469 14590 ts_tablet_manager.cc:1428] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:44:58.102559 14614 ts_tablet_manager.cc:1918] T 399736efc31546a380dabacf5cc2c7fd P 99f89e8e3392413ab58b7363975ea0ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:44:58.102640 14614 log.cc:1196] T 399736efc31546a380dabacf5cc2c7fd P 99f89e8e3392413ab58b7363975ea0ea: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/399736efc31546a380dabacf5cc2c7fd
I20260926 13:44:58.102910 14614 ts_tablet_manager.cc:1939] T 399736efc31546a380dabacf5cc2c7fd P 99f89e8e3392413ab58b7363975ea0ea: Deleting consensus metadata
I20260926 13:44:58.103739 14615 ts_tablet_manager.cc:1939] T 399736efc31546a380dabacf5cc2c7fd P ea8b31356cc244ee960bc10878467e8a: Deleting consensus metadata
I20260926 13:44:58.103941 14071 catalog_manager.cc:4841] TS 99f89e8e3392413ab58b7363975ea0ea (127.12.170.67:45747): tablet 399736efc31546a380dabacf5cc2c7fd (table test-table [id=e40b33010c27443dbe06aab2b8b964c7]) successfully deleted
I20260926 13:44:58.104590 14592 tablet_bootstrap.cc:492] T 812fd27453764152a89e22dbc85bdd1f P 99f89e8e3392413ab58b7363975ea0ea: No bootstrap required, opened a new log
I20260926 13:44:58.104663 14592 ts_tablet_manager.cc:1397] T 812fd27453764152a89e22dbc85bdd1f P 99f89e8e3392413ab58b7363975ea0ea: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260926 13:44:58.105037 14592 raft_consensus.cc:348] T 812fd27453764152a89e22dbc85bdd1f P 99f89e8e3392413ab58b7363975ea0ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } }
I20260926 13:44:58.105126 14592 raft_consensus.cc:374] T 812fd27453764152a89e22dbc85bdd1f P 99f89e8e3392413ab58b7363975ea0ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:58.105154 14592 raft_consensus.cc:729] T 812fd27453764152a89e22dbc85bdd1f P 99f89e8e3392413ab58b7363975ea0ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f89e8e3392413ab58b7363975ea0ea, State: Initialized, Role: FOLLOWER
I20260926 13:44:58.105209 14070 catalog_manager.cc:4841] TS ea8b31356cc244ee960bc10878467e8a (127.12.170.65:36001): tablet 399736efc31546a380dabacf5cc2c7fd (table test-table [id=e40b33010c27443dbe06aab2b8b964c7]) successfully deleted
I20260926 13:44:58.105253 14592 consensus_queue.cc:260] T 812fd27453764152a89e22dbc85bdd1f P 99f89e8e3392413ab58b7363975ea0ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } }
I20260926 13:44:58.105463 14592 ts_tablet_manager.cc:1428] T 812fd27453764152a89e22dbc85bdd1f P 99f89e8e3392413ab58b7363975ea0ea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260926 13:44:58.146106 14428 tablet.cc:2369] T 812fd27453764152a89e22dbc85bdd1f P 52709b0b591d404c9428fcb5155133e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:44:58.158723 14597 raft_consensus.cc:482] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:44:58.158845 14597 raft_consensus.cc:504] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } }
I20260926 13:44:58.159883 14597 leader_election.cc:288] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52709b0b591d404c9428fcb5155133e8 (127.12.170.66:43895), 99f89e8e3392413ab58b7363975ea0ea (127.12.170.67:45747)
I20260926 13:44:58.164115 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:58.160152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:38545 (local address 127.12.170.66:43895)
0926 13:44:58.160306 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:58.160311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:58.160420 (+ 109us) server_negotiation.cc:407] Connection header received
0926 13:44:58.160614 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:58.160618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:58.160667 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:58.160712 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:58.161119 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.161907 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.163204 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.163582 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.163704 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:58.163759 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:58.163940 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 13:44:58.164005 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 13:44:58.164130 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:58.160042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:43895 (local address 127.12.170.65:38545)
0926 13:44:58.160340 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:58.160353 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:58.160493 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:58.160842 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:58.160845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:58.160859 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:58.160960 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:58.160966 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.162010 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.162013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:58.163104 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:58.163112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.163726 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.163729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:58.163835 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:58.163855 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 13:44:58.163918 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 13:44:58.163988 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":61,"threads_started":1}
I20260926 13:44:58.164400 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:58.160449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:59447 (local address 127.12.170.67:45747)
0926 13:44:58.160571 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:58.160575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:58.160648 (+ 73us) server_negotiation.cc:407] Connection header received
0926 13:44:58.160821 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:58.160824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:58.160870 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:58.160944 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:44:58.161169 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.161647 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.163084 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.163720 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.163828 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:58.163878 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:58.164177 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 13:44:58.164267 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:44:58.164861 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:58.160102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:45747 (local address 127.12.170.65:59447)
0926 13:44:58.160577 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:58.160590 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:58.160715 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:58.160959 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:58.160961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:58.160974 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:44:58.161070 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:58.161076 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.161756 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.161759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:58.162965 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:58.162975 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.163937 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.163941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:58.164042 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:58.164064 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 13:44:58.164651 (+ 587us) client_negotiation.cc:240] Negotiation successful
0926 13:44:58.164731 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":101,"threads_started":1}
I20260926 13:44:58.164860 14381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812fd27453764152a89e22dbc85bdd1f" candidate_uuid: "ea8b31356cc244ee960bc10878467e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52709b0b591d404c9428fcb5155133e8" is_pre_election: true
I20260926 13:44:58.165084 14381 raft_consensus.cc:2454] T 812fd27453764152a89e22dbc85bdd1f P 52709b0b591d404c9428fcb5155133e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea8b31356cc244ee960bc10878467e8a in term 0.
I20260926 13:44:58.165359 14521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812fd27453764152a89e22dbc85bdd1f" candidate_uuid: "ea8b31356cc244ee960bc10878467e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f89e8e3392413ab58b7363975ea0ea" is_pre_election: true
I20260926 13:44:58.165485 14521 raft_consensus.cc:2454] T 812fd27453764152a89e22dbc85bdd1f P 99f89e8e3392413ab58b7363975ea0ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea8b31356cc244ee960bc10878467e8a in term 0.
I20260926 13:44:58.165452 14173 leader_election.cc:302] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52709b0b591d404c9428fcb5155133e8, ea8b31356cc244ee960bc10878467e8a; no voters:
I20260926 13:44:58.165746 14597 raft_consensus.cc:2781] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:44:58.165808 14597 raft_consensus.cc:482] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:44:58.165843 14597 raft_consensus.cc:3037] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:58.166742 14597 raft_consensus.cc:504] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } }
I20260926 13:44:58.167037 14597 leader_election.cc:288] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a [CANDIDATE]: Term 1 election: Requested vote from peers 52709b0b591d404c9428fcb5155133e8 (127.12.170.66:43895), 99f89e8e3392413ab58b7363975ea0ea (127.12.170.67:45747)
I20260926 13:44:58.167335 14381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812fd27453764152a89e22dbc85bdd1f" candidate_uuid: "ea8b31356cc244ee960bc10878467e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52709b0b591d404c9428fcb5155133e8"
I20260926 13:44:58.167354 14521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812fd27453764152a89e22dbc85bdd1f" candidate_uuid: "ea8b31356cc244ee960bc10878467e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f89e8e3392413ab58b7363975ea0ea"
I20260926 13:44:58.167447 14381 raft_consensus.cc:3037] T 812fd27453764152a89e22dbc85bdd1f P 52709b0b591d404c9428fcb5155133e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:58.167464 14521 raft_consensus.cc:3037] T 812fd27453764152a89e22dbc85bdd1f P 99f89e8e3392413ab58b7363975ea0ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:58.168287 14381 raft_consensus.cc:2454] T 812fd27453764152a89e22dbc85bdd1f P 52709b0b591d404c9428fcb5155133e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea8b31356cc244ee960bc10878467e8a in term 1.
I20260926 13:44:58.168535 14173 leader_election.cc:302] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52709b0b591d404c9428fcb5155133e8, ea8b31356cc244ee960bc10878467e8a; no voters:
I20260926 13:44:58.168558 14521 raft_consensus.cc:2454] T 812fd27453764152a89e22dbc85bdd1f P 99f89e8e3392413ab58b7363975ea0ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea8b31356cc244ee960bc10878467e8a in term 1.
I20260926 13:44:58.168702 14597 raft_consensus.cc:2781] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:58.168903 14597 raft_consensus.cc:686] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a [term 1 LEADER]: Becoming Leader. State: Replica: ea8b31356cc244ee960bc10878467e8a, State: Running, Role: LEADER
I20260926 13:44:58.169143 14597 consensus_queue.cc:237] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } }
I20260926 13:44:58.171047 14084 catalog_manager.cc:5495] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ea8b31356cc244ee960bc10878467e8a (127.12.170.65). New cstate: current_term: 1 leader_uuid: "ea8b31356cc244ee960bc10878467e8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } health_report { overall_health: UNKNOWN } } }
Sep 26 13:44:58 dist-test-slave-hc1k krb5kdc[14039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430297, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:44:58.224025 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:58.213484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:38031 (local address 127.0.0.1:38662)
0926 13:44:58.214997 (+ 1513us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:58.215012 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:58.215166 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:58.215489 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:58.215492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:58.216006 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:58.216146 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:58.216155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.217221 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.217224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:58.218084 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:58.218094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.218666 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.218670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:58.218765 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:58.219353 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:58.219368 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:58.220923 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:58.222549 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:58.222566 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:58.222571 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:58.222776 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:58.223045 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:58.223047 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:58.223049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:58.223143 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:58.223545 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:58.223550 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:58.223673 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 13:44:58.223784 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 13:44:58.223895 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1421,"thread_start_us":82,"threads_started":1}
I20260926 13:44:58.224128 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:58.213530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38662 (local address 127.12.170.126:38031)
0926 13:44:58.213658 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:58.213662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:58.215094 (+ 1432us) server_negotiation.cc:407] Connection header received
0926 13:44:58.215263 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:58.215266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:58.215304 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:58.215349 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:58.216284 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.217094 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.218188 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.218504 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.218631 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:58.221030 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:58.221034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:58.221036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:58.221060 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:58.222444 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:58.222852 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:58.222855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:58.222856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:58.222930 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:58.223228 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:58.223231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:58.223233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:58.223416 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:58.223500 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:58.223833 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 13:44:58.223958 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":54}
I20260926 13:44:58.227622 14084 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:38662:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 13:44:58.228386 14084 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:44:58.232728 14221 tablet_service.cc:1467] Processing CreateTablet for tablet daba1928c7ba4546b2612ec0d3b736b0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ae236349918b4ef4b8638b4ee33c86bb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:44:58.232867 14501 tablet_service.cc:1467] Processing CreateTablet for tablet daba1928c7ba4546b2612ec0d3b736b0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ae236349918b4ef4b8638b4ee33c86bb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:44:58.232960 14361 tablet_service.cc:1467] Processing CreateTablet for tablet daba1928c7ba4546b2612ec0d3b736b0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ae236349918b4ef4b8638b4ee33c86bb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:44:58.233109 14221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daba1928c7ba4546b2612ec0d3b736b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:58.233142 14501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daba1928c7ba4546b2612ec0d3b736b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:58.233299 14361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daba1928c7ba4546b2612ec0d3b736b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:58.235838 14590 tablet_bootstrap.cc:492] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a: Bootstrap starting.
I20260926 13:44:58.235855 14592 tablet_bootstrap.cc:492] T daba1928c7ba4546b2612ec0d3b736b0 P 99f89e8e3392413ab58b7363975ea0ea: Bootstrap starting.
I20260926 13:44:58.235980 14591 tablet_bootstrap.cc:492] T daba1928c7ba4546b2612ec0d3b736b0 P 52709b0b591d404c9428fcb5155133e8: Bootstrap starting.
I20260926 13:44:58.236727 14590 tablet_bootstrap.cc:654] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:58.236727 14592 tablet_bootstrap.cc:654] T daba1928c7ba4546b2612ec0d3b736b0 P 99f89e8e3392413ab58b7363975ea0ea: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:58.236776 14591 tablet_bootstrap.cc:654] T daba1928c7ba4546b2612ec0d3b736b0 P 52709b0b591d404c9428fcb5155133e8: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:58.238265 14592 tablet_bootstrap.cc:492] T daba1928c7ba4546b2612ec0d3b736b0 P 99f89e8e3392413ab58b7363975ea0ea: No bootstrap required, opened a new log
I20260926 13:44:58.238336 14592 ts_tablet_manager.cc:1397] T daba1928c7ba4546b2612ec0d3b736b0 P 99f89e8e3392413ab58b7363975ea0ea: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 13:44:58.238689 14592 raft_consensus.cc:348] T daba1928c7ba4546b2612ec0d3b736b0 P 99f89e8e3392413ab58b7363975ea0ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } }
I20260926 13:44:58.238785 14592 raft_consensus.cc:374] T daba1928c7ba4546b2612ec0d3b736b0 P 99f89e8e3392413ab58b7363975ea0ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:58.238813 14592 raft_consensus.cc:729] T daba1928c7ba4546b2612ec0d3b736b0 P 99f89e8e3392413ab58b7363975ea0ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f89e8e3392413ab58b7363975ea0ea, State: Initialized, Role: FOLLOWER
I20260926 13:44:58.238924 14592 consensus_queue.cc:260] T daba1928c7ba4546b2612ec0d3b736b0 P 99f89e8e3392413ab58b7363975ea0ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } }
I20260926 13:44:58.239151 14598 tablet_replica.cc:439] T daba1928c7ba4546b2612ec0d3b736b0 P 99f89e8e3392413ab58b7363975ea0ea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } } }
I20260926 13:44:58.239315 14598 tablet_replica.cc:442] T daba1928c7ba4546b2612ec0d3b736b0 P 99f89e8e3392413ab58b7363975ea0ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:44:58.239148 14592 ts_tablet_manager.cc:1428] T daba1928c7ba4546b2612ec0d3b736b0 P 99f89e8e3392413ab58b7363975ea0ea: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 13:44:58.239754 14591 tablet_bootstrap.cc:492] T daba1928c7ba4546b2612ec0d3b736b0 P 52709b0b591d404c9428fcb5155133e8: No bootstrap required, opened a new log
I20260926 13:44:58.239828 14590 tablet_bootstrap.cc:492] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a: No bootstrap required, opened a new log
I20260926 13:44:58.239831 14591 ts_tablet_manager.cc:1397] T daba1928c7ba4546b2612ec0d3b736b0 P 52709b0b591d404c9428fcb5155133e8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 13:44:58.239889 14590 ts_tablet_manager.cc:1397] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 13:44:58.240191 14591 raft_consensus.cc:348] T daba1928c7ba4546b2612ec0d3b736b0 P 52709b0b591d404c9428fcb5155133e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } }
I20260926 13:44:58.240289 14591 raft_consensus.cc:374] T daba1928c7ba4546b2612ec0d3b736b0 P 52709b0b591d404c9428fcb5155133e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:58.240250 14590 raft_consensus.cc:348] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } }
I20260926 13:44:58.240319 14591 raft_consensus.cc:729] T daba1928c7ba4546b2612ec0d3b736b0 P 52709b0b591d404c9428fcb5155133e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52709b0b591d404c9428fcb5155133e8, State: Initialized, Role: FOLLOWER
I20260926 13:44:58.240336 14590 raft_consensus.cc:374] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:58.240365 14590 raft_consensus.cc:729] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea8b31356cc244ee960bc10878467e8a, State: Initialized, Role: FOLLOWER
I20260926 13:44:58.240437 14591 consensus_queue.cc:260] T daba1928c7ba4546b2612ec0d3b736b0 P 52709b0b591d404c9428fcb5155133e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } }
I20260926 13:44:58.240461 14590 consensus_queue.cc:260] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } }
I20260926 13:44:58.240684 14591 ts_tablet_manager.cc:1428] T daba1928c7ba4546b2612ec0d3b736b0 P 52709b0b591d404c9428fcb5155133e8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 13:44:58.240684 14590 ts_tablet_manager.cc:1428] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:44:58.240710 14610 tablet_replica.cc:439] T daba1928c7ba4546b2612ec0d3b736b0 P 52709b0b591d404c9428fcb5155133e8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } } }
I20260926 13:44:58.240825 14610 tablet_replica.cc:442] T daba1928c7ba4546b2612ec0d3b736b0 P 52709b0b591d404c9428fcb5155133e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:44:58.240860 14597 tablet_replica.cc:439] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } } }
I20260926 13:44:58.240942 14597 tablet_replica.cc:442] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:44:58.270884 14597 raft_consensus.cc:482] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:44:58.270990 14597 raft_consensus.cc:504] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } }
I20260926 13:44:58.271358 14597 leader_election.cc:288] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52709b0b591d404c9428fcb5155133e8 (127.12.170.66:43895), 99f89e8e3392413ab58b7363975ea0ea (127.12.170.67:45747)
I20260926 13:44:58.271741 14381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daba1928c7ba4546b2612ec0d3b736b0" candidate_uuid: "ea8b31356cc244ee960bc10878467e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52709b0b591d404c9428fcb5155133e8" is_pre_election: true
I20260926 13:44:58.271747 14521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daba1928c7ba4546b2612ec0d3b736b0" candidate_uuid: "ea8b31356cc244ee960bc10878467e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f89e8e3392413ab58b7363975ea0ea" is_pre_election: true
I20260926 13:44:58.271880 14381 raft_consensus.cc:2454] T daba1928c7ba4546b2612ec0d3b736b0 P 52709b0b591d404c9428fcb5155133e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea8b31356cc244ee960bc10878467e8a in term 0.
I20260926 13:44:58.271880 14521 raft_consensus.cc:2454] T daba1928c7ba4546b2612ec0d3b736b0 P 99f89e8e3392413ab58b7363975ea0ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea8b31356cc244ee960bc10878467e8a in term 0.
I20260926 13:44:58.272126 14173 leader_election.cc:302] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52709b0b591d404c9428fcb5155133e8, ea8b31356cc244ee960bc10878467e8a; no voters:
I20260926 13:44:58.272297 14597 raft_consensus.cc:2781] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:44:58.272356 14597 raft_consensus.cc:482] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:44:58.272390 14597 raft_consensus.cc:3037] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:58.273274 14597 raft_consensus.cc:504] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } }
I20260926 13:44:58.273559 14597 leader_election.cc:288] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a [CANDIDATE]: Term 1 election: Requested vote from peers 52709b0b591d404c9428fcb5155133e8 (127.12.170.66:43895), 99f89e8e3392413ab58b7363975ea0ea (127.12.170.67:45747)
I20260926 13:44:58.273859 14521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daba1928c7ba4546b2612ec0d3b736b0" candidate_uuid: "ea8b31356cc244ee960bc10878467e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f89e8e3392413ab58b7363975ea0ea"
I20260926 13:44:58.273882 14381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daba1928c7ba4546b2612ec0d3b736b0" candidate_uuid: "ea8b31356cc244ee960bc10878467e8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52709b0b591d404c9428fcb5155133e8"
I20260926 13:44:58.273969 14521 raft_consensus.cc:3037] T daba1928c7ba4546b2612ec0d3b736b0 P 99f89e8e3392413ab58b7363975ea0ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:58.273975 14381 raft_consensus.cc:3037] T daba1928c7ba4546b2612ec0d3b736b0 P 52709b0b591d404c9428fcb5155133e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:44:58.274958 14381 raft_consensus.cc:2454] T daba1928c7ba4546b2612ec0d3b736b0 P 52709b0b591d404c9428fcb5155133e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea8b31356cc244ee960bc10878467e8a in term 1.
I20260926 13:44:58.274958 14521 raft_consensus.cc:2454] T daba1928c7ba4546b2612ec0d3b736b0 P 99f89e8e3392413ab58b7363975ea0ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea8b31356cc244ee960bc10878467e8a in term 1.
I20260926 13:44:58.275211 14173 leader_election.cc:302] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52709b0b591d404c9428fcb5155133e8, ea8b31356cc244ee960bc10878467e8a; no voters:
I20260926 13:44:58.275363 14597 raft_consensus.cc:2781] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:58.275446 14597 raft_consensus.cc:686] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a [term 1 LEADER]: Becoming Leader. State: Replica: ea8b31356cc244ee960bc10878467e8a, State: Running, Role: LEADER
I20260926 13:44:58.275566 14597 consensus_queue.cc:237] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } }
I20260926 13:44:58.276218 14621 tablet_replica.cc:439] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a: TxnStatusTablet state changed. Reason: New leader ea8b31356cc244ee960bc10878467e8a. Latest consensus state: current_term: 1 leader_uuid: "ea8b31356cc244ee960bc10878467e8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } } }
I20260926 13:44:58.276306 14621 tablet_replica.cc:442] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:44:58.276568 14633 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 13:44:58.276722 14633 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 13:44:58.277189 14084 catalog_manager.cc:5495] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a reported cstate change: term changed from 0 to 1, leader changed from <none> to ea8b31356cc244ee960bc10878467e8a (127.12.170.65). New cstate: current_term: 1 leader_uuid: "ea8b31356cc244ee960bc10878467e8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } health_report { overall_health: UNKNOWN } } }
I20260926 13:44:58.295782 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:58.292326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:36001 (local address 127.0.0.1:38238)
0926 13:44:58.292462 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:58.292478 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:58.292584 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:58.292864 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:58.292866 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:58.292873 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:44:58.292967 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:58.292972 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.293650 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.293654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:58.294275 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:58.294283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.294683 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.294686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:58.294782 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:58.294822 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:44:58.295469 (+ 647us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:44:58.295485 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 13:44:58.295561 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 13:44:58.295630 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 13:44:58.295792 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:58.292443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38238 (local address 127.12.170.65:36001)
0926 13:44:58.292546 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:58.292551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:58.292564 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:44:58.292712 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:58.292715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:58.292769 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:58.292844 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:44:58.293067 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.293555 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.294380 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.294594 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.294675 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:58.294948 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:44:58.295391 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:44:58.295483 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:58.295593 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 13:44:58.295673 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 13:44:58.298233 14521 raft_consensus.cc:1261] T daba1928c7ba4546b2612ec0d3b736b0 P 99f89e8e3392413ab58b7363975ea0ea [term 1 FOLLOWER]: Refusing update from remote peer ea8b31356cc244ee960bc10878467e8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:44:58.298399 14381 raft_consensus.cc:1261] T daba1928c7ba4546b2612ec0d3b736b0 P 52709b0b591d404c9428fcb5155133e8 [term 1 FOLLOWER]: Refusing update from remote peer ea8b31356cc244ee960bc10878467e8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:44:58.298916 14621 consensus_queue.cc:1035] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:44:58.299103 14597 consensus_queue.cc:1035] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:44:58.300578 14610 tablet_replica.cc:439] T daba1928c7ba4546b2612ec0d3b736b0 P 52709b0b591d404c9428fcb5155133e8: TxnStatusTablet state changed. Reason: New leader ea8b31356cc244ee960bc10878467e8a. Latest consensus state: current_term: 1 leader_uuid: "ea8b31356cc244ee960bc10878467e8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } } }
I20260926 13:44:58.300689 14610 tablet_replica.cc:442] T daba1928c7ba4546b2612ec0d3b736b0 P 52709b0b591d404c9428fcb5155133e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:44:58.300740 14598 tablet_replica.cc:439] T daba1928c7ba4546b2612ec0d3b736b0 P 99f89e8e3392413ab58b7363975ea0ea: TxnStatusTablet state changed. Reason: New leader ea8b31356cc244ee960bc10878467e8a. Latest consensus state: current_term: 1 leader_uuid: "ea8b31356cc244ee960bc10878467e8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } } }
I20260926 13:44:58.300863 14598 tablet_replica.cc:442] T daba1928c7ba4546b2612ec0d3b736b0 P 99f89e8e3392413ab58b7363975ea0ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:44:58.302971 14621 tablet_replica.cc:439] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea8b31356cc244ee960bc10878467e8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } } }
I20260926 13:44:58.303076 14621 tablet_replica.cc:442] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:44:58.303190 14621 tablet_replica.cc:439] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea8b31356cc244ee960bc10878467e8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } } }
I20260926 13:44:58.303256 14621 tablet_replica.cc:442] T daba1928c7ba4546b2612ec0d3b736b0 P ea8b31356cc244ee960bc10878467e8a: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:44:58.303565 14605 mvcc.cc:205] Tried to move back new op lower bound from 7333602501825323008 to 7333602501737996288. Current Snapshot: MvccSnapshot[applied={T|T < 7333602501825323008}]
I20260926 13:44:58.303731 14598 tablet_replica.cc:439] T daba1928c7ba4546b2612ec0d3b736b0 P 99f89e8e3392413ab58b7363975ea0ea: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea8b31356cc244ee960bc10878467e8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } } }
I20260926 13:44:58.303815 14598 tablet_replica.cc:442] T daba1928c7ba4546b2612ec0d3b736b0 P 99f89e8e3392413ab58b7363975ea0ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:44:58.308249 14610 tablet_replica.cc:439] T daba1928c7ba4546b2612ec0d3b736b0 P 52709b0b591d404c9428fcb5155133e8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea8b31356cc244ee960bc10878467e8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea8b31356cc244ee960bc10878467e8a" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 36001 } } peers { permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 } } peers { permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 } } }
I20260926 13:44:58.308348 14610 tablet_replica.cc:442] T daba1928c7ba4546b2612ec0d3b736b0 P 52709b0b591d404c9428fcb5155133e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:44:58.310439 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:58.307012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38246 (local address 127.12.170.65:36001)
0926 13:44:58.307118 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:58.307122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:58.307180 (+ 58us) server_negotiation.cc:407] Connection header received
0926 13:44:58.307336 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:58.307339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:58.307380 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:58.307440 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:44:58.307770 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.308439 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.309446 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.309726 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.309815 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:58.310066 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:44:58.310128 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:44:58.310183 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:58.310273 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 13:44:58.310318 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 13:44:58.319854 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:58.316300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:59211 (local address 127.12.170.65:36001)
0926 13:44:58.316395 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:58.316399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:58.316589 (+ 190us) server_negotiation.cc:407] Connection header received
0926 13:44:58.316745 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:58.316748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:58.316811 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:58.316880 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:44:58.317173 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.317901 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.318804 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.319133 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.319227 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:58.319435 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:44:58.319502 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:44:58.319564 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:58.319676 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 13:44:58.319713 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":35}
I20260926 13:44:58.320254 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:58.316237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:36001 (local address 127.12.170.65:59211)
0926 13:44:58.316505 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:58.316517 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:58.316634 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:58.316915 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:58.316918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:58.316926 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:44:58.317054 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:58.317061 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.317994 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.317997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:58.318707 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:58.318717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.319226 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.319228 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:58.319305 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:58.319340 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:44:58.319575 (+ 235us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:44:58.319579 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:44:58.319632 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 13:44:58.319719 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":1,"thread_start_us":97,"threads_started":1}
I20260926 13:44:58.325860 14521 raft_consensus.cc:1261] T 812fd27453764152a89e22dbc85bdd1f P 99f89e8e3392413ab58b7363975ea0ea [term 1 FOLLOWER]: Refusing update from remote peer ea8b31356cc244ee960bc10878467e8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:44:58.326049 14380 raft_consensus.cc:1261] T 812fd27453764152a89e22dbc85bdd1f P 52709b0b591d404c9428fcb5155133e8 [term 1 FOLLOWER]: Refusing update from remote peer ea8b31356cc244ee960bc10878467e8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:44:58.326228 14621 consensus_queue.cc:1035] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f89e8e3392413ab58b7363975ea0ea" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 45747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:44:58.326390 14638 consensus_queue.cc:1035] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "52709b0b591d404c9428fcb5155133e8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:44:58.363595 14083 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:44:58.363777 14083 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:44:58.365490 14083 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 486bfe16915e479ea3d6542a0ab35e49: Sending DeleteTablet for 3 replicas of tablet 812fd27453764152a89e22dbc85bdd1f
I20260926 13:44:58.365984 14361 tablet_service.cc:1514] Processing DeleteTablet for tablet 812fd27453764152a89e22dbc85bdd1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:44:58 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:59502
I20260926 13:44:58.366078 14501 tablet_service.cc:1514] Processing DeleteTablet for tablet 812fd27453764152a89e22dbc85bdd1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:44:58 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:48894
I20260926 13:44:58.366163 14613 tablet_replica.cc:330] T 812fd27453764152a89e22dbc85bdd1f P 52709b0b591d404c9428fcb5155133e8: stopping tablet replica
I20260926 13:44:58.366212 14614 tablet_replica.cc:330] T 812fd27453764152a89e22dbc85bdd1f P 99f89e8e3392413ab58b7363975ea0ea: stopping tablet replica
I20260926 13:44:58.366261 14613 raft_consensus.cc:2229] T 812fd27453764152a89e22dbc85bdd1f P 52709b0b591d404c9428fcb5155133e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:44:58.366291 14614 raft_consensus.cc:2229] T 812fd27453764152a89e22dbc85bdd1f P 99f89e8e3392413ab58b7363975ea0ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:44:58.366364 14613 raft_consensus.cc:2258] T 812fd27453764152a89e22dbc85bdd1f P 52709b0b591d404c9428fcb5155133e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:44:58.366370 14614 raft_consensus.cc:2258] T 812fd27453764152a89e22dbc85bdd1f P 99f89e8e3392413ab58b7363975ea0ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:44:58.366514 14220 tablet_service.cc:1514] Processing DeleteTablet for tablet 812fd27453764152a89e22dbc85bdd1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:44:58 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:38230
I20260926 13:44:58.366664 14615 tablet_replica.cc:330] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a: stopping tablet replica
I20260926 13:44:58.366750 14615 raft_consensus.cc:2229] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:44:58.366997 14615 raft_consensus.cc:2258] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:44:58.367373 14614 ts_tablet_manager.cc:1905] T 812fd27453764152a89e22dbc85bdd1f P 99f89e8e3392413ab58b7363975ea0ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:44:58.367635 14615 ts_tablet_manager.cc:1905] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:44:58.370078 14615 ts_tablet_manager.cc:1918] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:44:58.370077 14614 ts_tablet_manager.cc:1918] T 812fd27453764152a89e22dbc85bdd1f P 99f89e8e3392413ab58b7363975ea0ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:44:58.370143 14614 log.cc:1196] T 812fd27453764152a89e22dbc85bdd1f P 99f89e8e3392413ab58b7363975ea0ea: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/812fd27453764152a89e22dbc85bdd1f
I20260926 13:44:58.370143 14615 log.cc:1196] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-0/wal/wals/812fd27453764152a89e22dbc85bdd1f
I20260926 13:44:58.370371 14614 ts_tablet_manager.cc:1939] T 812fd27453764152a89e22dbc85bdd1f P 99f89e8e3392413ab58b7363975ea0ea: Deleting consensus metadata
I20260926 13:44:58.370370 14615 ts_tablet_manager.cc:1939] T 812fd27453764152a89e22dbc85bdd1f P ea8b31356cc244ee960bc10878467e8a: Deleting consensus metadata
I20260926 13:44:58.371099 14070 catalog_manager.cc:4841] TS ea8b31356cc244ee960bc10878467e8a (127.12.170.65:36001): tablet 812fd27453764152a89e22dbc85bdd1f (table test-table [id=8857f8b937d442ed89ea4ae0d98dd6b5]) successfully deleted
I20260926 13:44:58.371150 14071 catalog_manager.cc:4841] TS 99f89e8e3392413ab58b7363975ea0ea (127.12.170.67:45747): tablet 812fd27453764152a89e22dbc85bdd1f (table test-table [id=8857f8b937d442ed89ea4ae0d98dd6b5]) successfully deleted
I20260926 13:44:58.371418 14613 ts_tablet_manager.cc:1905] T 812fd27453764152a89e22dbc85bdd1f P 52709b0b591d404c9428fcb5155133e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:44:58.373755 14613 ts_tablet_manager.cc:1918] T 812fd27453764152a89e22dbc85bdd1f P 52709b0b591d404c9428fcb5155133e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:44:58.373821 14613 log.cc:1196] T 812fd27453764152a89e22dbc85bdd1f P 52709b0b591d404c9428fcb5155133e8: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790430291939677-12969-0/minicluster-data/ts-1/wal/wals/812fd27453764152a89e22dbc85bdd1f
I20260926 13:44:58.374011 14613 ts_tablet_manager.cc:1939] T 812fd27453764152a89e22dbc85bdd1f P 52709b0b591d404c9428fcb5155133e8: Deleting consensus metadata
I20260926 13:44:58.374538 14069 catalog_manager.cc:4841] TS 52709b0b591d404c9428fcb5155133e8 (127.12.170.66:43895): tablet 812fd27453764152a89e22dbc85bdd1f (table test-table [id=8857f8b937d442ed89ea4ae0d98dd6b5]) successfully deleted
Sep 26 13:44:58 dist-test-slave-hc1k krb5kdc[14039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430297, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.170.65@KRBTEST.COM
I20260926 13:44:58.381419 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:58.367269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38260 (local address 127.12.170.65:36001)
0926 13:44:58.370611 (+ 3342us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:58.370615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:58.371623 (+ 1008us) server_negotiation.cc:407] Connection header received
0926 13:44:58.371763 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:58.371766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:58.371814 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:58.371892 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:58.372682 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.373381 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.374475 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.374774 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.374852 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:58.378095 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:58.378104 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:58.378114 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:58.378135 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:58.379644 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:58.380161 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:58.380167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:58.380169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:58.380262 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:58.380507 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:58.380510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:58.380512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:58.380751 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:58.380945 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:58.381163 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 13:44:58.381268 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3219}
W20260926 13:44:58.381815 14281 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38260
I20260926 13:44:58.390166 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:58.383250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38676 (local address 127.12.170.126:38031)
0926 13:44:58.383359 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:58.383362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:58.383418 (+ 56us) server_negotiation.cc:407] Connection header received
0926 13:44:58.383492 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:58.383495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:58.383541 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:58.383618 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:58.384152 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.384913 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.385693 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:58.386025 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:58.386134 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:58.387276 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:58.387280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:58.387283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:58.387305 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:58.388747 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:58.389102 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:58.389106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:58.389108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:58.389196 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:58.389409 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:58.389412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:58.389414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:58.389601 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:58.389719 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:58.389904 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 13:44:58.390014 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 13:44:58.390503 14083 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38676
I20260926 13:44:58.391794 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 14156
I20260926 13:44:58.398813 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 14296
I20260926 13:44:58.404078 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 14436
I20260926 13:44:58.408595 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 14052
2026-09-26T13:44:58Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2438 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:44:58 dist-test-slave-hc1k krb5kdc[14652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:44:58 dist-test-slave-hc1k krb5kdc[14652](info): set up 2 sockets
Sep 26 13:44:58 dist-test-slave-hc1k krb5kdc[14652](info): commencing operation
krb5kdc: starting...
W20260926 13:44:59.359241 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:44:59 dist-test-slave-hc1k krb5kdc[14652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:44:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:44:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:44:59.466713 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:46601
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:35477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:46601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:59.568239 14665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:59.568454 14665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:59.568490 14665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:59.572252 14665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:44:59.572291 14665 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 13:44:59.572305 14665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:59.572321 14665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:59.572335 14665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:44:59.572355 14665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:44:59.577169 14665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:35477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:46601
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:46601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:59.577991 14665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:59.578890 14665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:59.586557 14670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:59.586606 14665 server_base.cc:1034] running on GCE node
W20260926 13:44:59.586529 14671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:59.586776 14673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:59.587250 14665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:59.588299 14665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:59.589457 14665 hybrid_clock.cc:648] HybridClock initialized: now 1790430299589445 us; error 23 us; skew 500 ppm
Sep 26 13:44:59 dist-test-slave-hc1k krb5kdc[14652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430299, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:59.591955 14665 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:44:59.592686 14665 webserver.cc:460] Webserver started at http://127.12.170.126:37917/ using document root <none> and password file <none>
I20260926 13:44:59.593142 14665 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:59.593194 14665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:59.593341 14665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:59.595112 14665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "ef95c2a78e3449aea15ee4a910da5aa4"
format_stamp: "Formatted at 2026-09-26 13:44:59 on dist-test-slave-hc1k"
server_key: "570f87e31a742ad2c81fc9584ec4949e"
server_key_iv: "83bce7700a3bc310cbec54a4906a4013"
server_key_version: "encryptionkey@0"
I20260926 13:44:59.595564 14665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "ef95c2a78e3449aea15ee4a910da5aa4"
format_stamp: "Formatted at 2026-09-26 13:44:59 on dist-test-slave-hc1k"
server_key: "570f87e31a742ad2c81fc9584ec4949e"
server_key_iv: "83bce7700a3bc310cbec54a4906a4013"
server_key_version: "encryptionkey@0"
I20260926 13:44:59.598816 14665 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260926 13:44:59.600924 14680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:59.602053 14665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:44:59.602164 14665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "ef95c2a78e3449aea15ee4a910da5aa4"
format_stamp: "Formatted at 2026-09-26 13:44:59 on dist-test-slave-hc1k"
server_key: "570f87e31a742ad2c81fc9584ec4949e"
server_key_iv: "83bce7700a3bc310cbec54a4906a4013"
server_key_version: "encryptionkey@0"
I20260926 13:44:59.602267 14665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:59.620875 14665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:59.623433 14665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:59.623615 14665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:59.631341 14665 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:46601
I20260926 13:44:59.631366 14742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:46601 every 8 connection(s)
I20260926 13:44:59.632223 14665 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:44:59.634686 14743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:44:59.639613 14743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4: Bootstrap starting.
I20260926 13:44:59.641453 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 14665
I20260926 13:44:59.641562 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:44:59.641819 12969 external_mini_cluster.cc:1428] Setting key 7d25adc9305e00f8e235e37264eebeb4
I20260926 13:44:59.641889 14743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4: Neither blocks nor log segments found. Creating new log.
I20260926 13:44:59.642741 14743 log.cc:824] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4: Log is configured to *not* fsync() on all Append() calls
I20260926 13:44:59.644618 14743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4: No bootstrap required, opened a new log
I20260926 13:44:59.648686 14743 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef95c2a78e3449aea15ee4a910da5aa4" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 46601 } }
I20260926 13:44:59.648938 14743 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:44:59.649030 14743 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef95c2a78e3449aea15ee4a910da5aa4, State: Initialized, Role: FOLLOWER
I20260926 13:44:59.649501 14743 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef95c2a78e3449aea15ee4a910da5aa4" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 46601 } }
I20260926 13:44:59.649655 14743 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:44:59.649721 14743 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:44:59.649816 14743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 13:44:59 dist-test-slave-hc1k krb5kdc[14652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:44:59.650820 14743 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef95c2a78e3449aea15ee4a910da5aa4" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 46601 } }
I20260926 13:44:59.651134 14743 leader_election.cc:302] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef95c2a78e3449aea15ee4a910da5aa4; no voters:
I20260926 13:44:59.651427 14743 leader_election.cc:288] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:44:59.651539 14748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:44:59.651759 14748 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4 [term 1 LEADER]: Becoming Leader. State: Replica: ef95c2a78e3449aea15ee4a910da5aa4, State: Running, Role: LEADER
I20260926 13:44:59.652066 14748 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef95c2a78e3449aea15ee4a910da5aa4" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 46601 } }
I20260926 13:44:59.652575 14743 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:44:59.653267 14748 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef95c2a78e3449aea15ee4a910da5aa4. Latest consensus state: current_term: 1 leader_uuid: "ef95c2a78e3449aea15ee4a910da5aa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef95c2a78e3449aea15ee4a910da5aa4" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 46601 } } }
I20260926 13:44:59.653390 14748 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:59.653746 14749 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef95c2a78e3449aea15ee4a910da5aa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef95c2a78e3449aea15ee4a910da5aa4" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 46601 } } }
I20260926 13:44:59.653895 14749 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4 [sys.catalog]: This master's current role is: LEADER
I20260926 13:44:59.656713 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:59.642852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38458 (local address 127.12.170.126:46601)
0926 13:44:59.643260 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:59.643278 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:59.643320 (+ 42us) server_negotiation.cc:407] Connection header received
0926 13:44:59.643977 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:59.643995 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:59.644291 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:59.644553 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:59.645446 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:59.646389 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:59.647592 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:59.648059 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:59.648168 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:59.650982 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:59.650989 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:59.651006 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:59.651034 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:59.653070 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:59.653622 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:59.653628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:59.653633 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:59.653768 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:59.654133 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:59.654136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:59.654138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:59.654616 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:59.655413 (+ 797us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:59.655565 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 13:44:59.655860 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":151,"threads_started":1}
W20260926 13:44:59.657181 14763 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:44:59.657249 14763 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:44:59.657301 14757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:44:59.658015 14757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:44:59.663090 14757 catalog_manager.cc:1347] Generated new cluster ID: b72cf1a06c2c410189c1f3582ec10aa5
I20260926 13:44:59.663141 14757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:44:59.669100 14757 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:44:59.669878 14757 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:44:59.678253 14757 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4: Generated new TSK 0
I20260926 13:44:59.678848 14757 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.170.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
I20260926 13:44:59.704209 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:0
--local_ip_for_outbound_sockets=127.12.170.65
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:46601
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:35477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:44:59.807412 14769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:44:59.807633 14769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:44:59.807705 14769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:44:59.811605 14769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:44:59.811673 14769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:44:59.811748 14769 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 13:44:59.811769 14769 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:44:59.811805 14769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:44:59.816745 14769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:35477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.170.126:46601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:44:59.817641 14769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:44:59.818601 14769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:44:59.827260 14774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:44:59.827273 14777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:59.827333 14769 server_base.cc:1034] running on GCE node
W20260926 13:44:59.827279 14775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:44:59.827880 14769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:44:59.828425 14769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:44:59.829583 14769 hybrid_clock.cc:648] HybridClock initialized: now 1790430299829564 us; error 36 us; skew 500 ppm
Sep 26 13:44:59 dist-test-slave-hc1k krb5kdc[14652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430299, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:44:59.832013 14769 init.cc:377] Logged in from keytab as kudu/127.12.170.65@KRBTEST.COM (short username kudu)
I20260926 13:44:59.832746 14769 webserver.cc:460] Webserver started at http://127.12.170.65:45417/ using document root <none> and password file <none>
I20260926 13:44:59.833230 14769 fs_manager.cc:359] Metadata directory not provided
I20260926 13:44:59.833283 14769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:44:59.833467 14769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:44:59.835296 14769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-0/data/instance:
uuid: "a1299c2a5fc0416f8d90363abdb436e2"
format_stamp: "Formatted at 2026-09-26 13:44:59 on dist-test-slave-hc1k"
server_key: "c231d0ac4470b285c10e65c424224c09"
server_key_iv: "b752ec190198e08401c73d948340ec63"
server_key_version: "encryptionkey@0"
I20260926 13:44:59.835821 14769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance:
uuid: "a1299c2a5fc0416f8d90363abdb436e2"
format_stamp: "Formatted at 2026-09-26 13:44:59 on dist-test-slave-hc1k"
server_key: "c231d0ac4470b285c10e65c424224c09"
server_key_iv: "b752ec190198e08401c73d948340ec63"
server_key_version: "encryptionkey@0"
I20260926 13:44:59.839071 14769 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260926 13:44:59.841231 14784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:59.842383 14769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 13:44:59.842481 14769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "a1299c2a5fc0416f8d90363abdb436e2"
format_stamp: "Formatted at 2026-09-26 13:44:59 on dist-test-slave-hc1k"
server_key: "c231d0ac4470b285c10e65c424224c09"
server_key_iv: "b752ec190198e08401c73d948340ec63"
server_key_version: "encryptionkey@0"
I20260926 13:44:59.842567 14769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:44:59.860759 14769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:44:59.863348 14769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:44:59.863495 14769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:44:59.864765 14769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:44:59.864810 14769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:44:59.864858 14769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:44:59.864886 14769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 13:44:59 dist-test-slave-hc1k krb5kdc[14652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430299, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:44:59.874401 14769 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:41243
I20260926 13:44:59.874680 14899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:41243 every 8 connection(s)
I20260926 13:44:59.875362 14769 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:44:59.877616 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:59.865599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:60835 (local address 127.12.170.126:46601)
0926 13:44:59.865714 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:59.865717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:59.866667 (+ 950us) server_negotiation.cc:407] Connection header received
0926 13:44:59.867750 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:59.867753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:59.867814 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:59.867910 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:59.869197 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:59.870002 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:59.871320 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:59.871573 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:59.871696 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:59.874077 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:59.874080 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:59.874082 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:59.874108 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:59.875753 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:59.876417 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:59.876420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:59.876421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:59.876508 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:59.876787 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:59.876790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:59.876791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:59.876955 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:59.877029 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:59.877402 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 13:44:59.877498 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 13:44:59.878291 14793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:59.866059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:46601 (local address 127.12.170.65:60835)
0926 13:44:59.866516 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:59.866556 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:59.867500 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:59.868111 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:59.868120 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:59.868460 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:59.869022 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:59.869065 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:59.870151 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:59.870155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:59.871184 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:59.871194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:59.871757 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:59.871762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:59.871917 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:59.872439 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:59.872456 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:59.873902 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:59.875978 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:59.875983 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:59.875997 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:59.876281 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:59.876592 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:59.876594 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:59.876596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:59.876715 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:59.877027 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:59.877031 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:59.877228 (+ 197us) client_negotiation.cc:769] Sending connection context
0926 13:44:59.877367 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 13:44:59.877596 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":119,"threads_started":1}
I20260926 13:44:59.879369 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 14769
I20260926 13:44:59.879397 14900 heartbeater.cc:347] Connected to a master server at 127.12.170.126:46601
I20260926 13:44:59.879483 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance
I20260926 13:44:59.879521 14900 heartbeater.cc:464] Registering TS with master...
I20260926 13:44:59.879803 12969 external_mini_cluster.cc:1428] Setting key e81bfa866e5a98afeb244fee0e086623
I20260926 13:44:59.880007 14900 heartbeater.cc:511] Master 127.12.170.126:46601 requested a full tablet report, sending...
I20260926 13:44:59.881460 14696 ts_manager.cc:194] Registered new tserver with Master: a1299c2a5fc0416f8d90363abdb436e2 (127.12.170.65:41243)
I20260926 13:44:59.882807 14696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.65@KRBTEST.COM'} at 127.12.170.65:60835
I20260926 13:44:59.888358 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:59.881481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:46509 (local address 127.12.170.126:46601)
0926 13:44:59.881581 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 13:44:59.881585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:44:59.881876 (+ 291us) server_negotiation.cc:407] Connection header received
0926 13:44:59.881962 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:44:59.881965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:44:59.882005 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:44:59.882066 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:44:59.882519 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:59.883052 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:59.884048 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:44:59.884358 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:44:59.884455 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:59.885477 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:44:59.885480 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:44:59.885483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:44:59.885505 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:44:59.886907 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:59.887325 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:59.887329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:59.887330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:59.887401 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:44:59.887641 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:44:59.887644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:44:59.887646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:44:59.887815 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:44:59.887894 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 13:44:59.888136 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 13:44:59.888225 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 13:44:59.888368 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:44:59.881563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:46601 (local address 127.12.170.65:46509)
0926 13:44:59.881802 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 13:44:59.881814 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:44:59.881894 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:44:59.882169 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:44:59.882171 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:44:59.882335 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:44:59.882419 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:59.882423 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:59.883162 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:59.883165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:59.883927 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:44:59.883937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:44:59.884469 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:44:59.884473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:44:59.884567 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:44:59.885056 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:44:59.885067 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:44:59.885359 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:44:59.887018 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:59.887021 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:59.887023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:59.887240 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:59.887476 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:44:59.887479 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:44:59.887481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:44:59.887556 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:44:59.887895 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:44:59.887898 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:44:59.887980 (+ 82us) client_negotiation.cc:769] Sending connection context
0926 13:44:59.888063 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 13:44:59.888185 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.12.170.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
I20260926 13:44:59.905829 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:0
--local_ip_for_outbound_sockets=127.12.170.66
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:46601
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:35477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:00.008379 14909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:00.008622 14909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:00.008687 14909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:00.012586 14909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:00.012636 14909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:00.012691 14909 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 13:45:00.012717 14909 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:45:00.012763 14909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:45:00.017733 14909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:35477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.170.126:46601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:00.018631 14909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:00.019544 14909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:00.028156 14915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:00.028261 14914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:00.028267 14909 server_base.cc:1034] running on GCE node
W20260926 13:45:00.028168 14917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:00.028810 14909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:00.029361 14909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:00.030505 14909 hybrid_clock.cc:648] HybridClock initialized: now 1790430300030491 us; error 24 us; skew 500 ppm
Sep 26 13:45:00 dist-test-slave-hc1k krb5kdc[14652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430300, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:00.033077 14909 init.cc:377] Logged in from keytab as kudu/127.12.170.66@KRBTEST.COM (short username kudu)
I20260926 13:45:00.033819 14909 webserver.cc:460] Webserver started at http://127.12.170.66:41995/ using document root <none> and password file <none>
I20260926 13:45:00.034310 14909 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:00.034369 14909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:00.034544 14909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:00.036440 14909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-1/data/instance:
uuid: "e41140ad3adc462ea3e3b92afebdf016"
format_stamp: "Formatted at 2026-09-26 13:45:00 on dist-test-slave-hc1k"
server_key: "32ec0b366a89c152857a10ec3b7a8d4e"
server_key_iv: "df3f7f493c6ccff68333d2046397d133"
server_key_version: "encryptionkey@0"
I20260926 13:45:00.036942 14909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance:
uuid: "e41140ad3adc462ea3e3b92afebdf016"
format_stamp: "Formatted at 2026-09-26 13:45:00 on dist-test-slave-hc1k"
server_key: "32ec0b366a89c152857a10ec3b7a8d4e"
server_key_iv: "df3f7f493c6ccff68333d2046397d133"
server_key_version: "encryptionkey@0"
I20260926 13:45:00.040287 14909 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:45:00.042414 14924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:00.043502 14909 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 13:45:00.043601 14909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "e41140ad3adc462ea3e3b92afebdf016"
format_stamp: "Formatted at 2026-09-26 13:45:00 on dist-test-slave-hc1k"
server_key: "32ec0b366a89c152857a10ec3b7a8d4e"
server_key_iv: "df3f7f493c6ccff68333d2046397d133"
server_key_version: "encryptionkey@0"
I20260926 13:45:00.043711 14909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:00.064512 14909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:00.067188 14909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:00.067353 14909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:00.068706 14909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:00.068753 14909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:00.068804 14909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:00.068833 14909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 13:45:00 dist-test-slave-hc1k krb5kdc[14652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430300, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:00.078444 14909 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:43493
I20260926 13:45:00.078770 15039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:43493 every 8 connection(s)
I20260926 13:45:00.079372 14909 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:45:00.080938 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 14909
I20260926 13:45:00.081036 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance
I20260926 13:45:00.081338 12969 external_mini_cluster.cc:1428] Setting key 18c6211c40a3eb78af503ac61150a764
I20260926 13:45:00.082530 14933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.070077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:46601 (local address 127.12.170.66:36841)
0926 13:45:00.070517 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:00.070556 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:00.071474 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:00.072009 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:00.072016 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:00.072392 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:00.072955 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.072975 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.073995 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.073998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.074970 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.074979 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.075453 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.075457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.075618 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.076160 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:00.076176 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:00.077650 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:00.079519 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:00.079524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:00.079538 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:00.079894 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:00.080182 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:00.080186 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:00.080188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:00.080342 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:00.080826 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:00.080832 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:00.081116 (+ 284us) client_negotiation.cc:769] Sending connection context
0926 13:45:00.081303 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 13:45:00.081596 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":119,"threads_started":1}
I20260926 13:45:00.083091 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.069555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:36841 (local address 127.12.170.126:46601)
0926 13:45:00.069667 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:00.069671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:00.070716 (+ 1045us) server_negotiation.cc:407] Connection header received
0926 13:45:00.071706 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:00.071709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:00.071766 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:00.071841 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:00.073089 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.073881 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.075080 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.075311 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.075397 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.077804 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:00.077807 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:00.077809 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:00.077832 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:00.079397 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:00.080005 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:00.080008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:00.080010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:00.080098 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:00.080459 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:00.080462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:00.080464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:00.080686 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:00.080776 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:00.082865 (+ 2089us) server_negotiation.cc:299] Negotiation successful
0926 13:45:00.082958 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 13:45:00.084739 15040 heartbeater.cc:347] Connected to a master server at 127.12.170.126:46601
I20260926 13:45:00.084877 15040 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:00.085374 15040 heartbeater.cc:511] Master 127.12.170.126:46601 requested a full tablet report, sending...
I20260926 13:45:00.086321 14697 ts_manager.cc:194] Registered new tserver with Master: e41140ad3adc462ea3e3b92afebdf016 (127.12.170.66:43493)
I20260926 13:45:00.087002 14697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.66@KRBTEST.COM'} at 127.12.170.66:36841
WARNING: no policy specified for kudu/127.12.170.67@KRBTEST.COM; defaulting to no policy
I20260926 13:45:00.092692 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.085849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:59203 (local address 127.12.170.126:46601)
0926 13:45:00.085947 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:00.085950 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:00.086119 (+ 169us) server_negotiation.cc:407] Connection header received
0926 13:45:00.086238 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:00.086242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:00.086287 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:00.086352 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:00.086730 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.087310 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.088394 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.088721 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.088824 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.089794 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:00.089798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:00.089801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:00.089826 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:00.091157 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:00.091501 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:00.091503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:00.091505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:00.091576 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:00.091845 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:00.091848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:00.091850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:00.092146 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:00.092223 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:00.092494 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 13:45:00.092575 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 13:45:00.092713 15047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.085778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:46601 (local address 127.12.170.66:59203)
0926 13:45:00.086043 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:00.086056 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:00.086141 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:00.086363 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:00.086366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:00.086547 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:00.086631 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.086636 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.087407 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.087411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.088271 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.088280 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.088838 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.088842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.088932 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.089355 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:00.089365 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:00.089647 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:00.091248 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:00.091251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:00.091252 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:00.091423 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:00.091640 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:00.091642 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:00.091644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:00.091759 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:00.092283 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:00.092286 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:00.092362 (+ 76us) client_negotiation.cc:769] Sending connection context
0926 13:45:00.092456 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 13:45:00.092574 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":81,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
I20260926 13:45:00.105788 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:0
--local_ip_for_outbound_sockets=127.12.170.67
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:46601
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:35477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:00.211896 15049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:00.212138 15049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:00.212181 15049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:00.216048 15049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:00.216095 15049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:00.216131 15049 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 13:45:00.216144 15049 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:45:00.216172 15049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:45:00.221164 15049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:35477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.170.126:46601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:00.222072 15049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:00.223004 15049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:00.232309 15054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:00.232357 15057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:00.232316 15055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:00.232568 15049 server_base.cc:1034] running on GCE node
I20260926 13:45:00.232941 15049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:00.233492 15049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:00.234647 15049 hybrid_clock.cc:648] HybridClock initialized: now 1790430300234633 us; error 30 us; skew 500 ppm
Sep 26 13:45:00 dist-test-slave-hc1k krb5kdc[14652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430300, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:00.237365 15049 init.cc:377] Logged in from keytab as kudu/127.12.170.67@KRBTEST.COM (short username kudu)
I20260926 13:45:00.238190 15049 webserver.cc:460] Webserver started at http://127.12.170.67:43333/ using document root <none> and password file <none>
I20260926 13:45:00.238686 15049 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:00.238741 15049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:00.238917 15049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:00.240860 15049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-2/data/instance:
uuid: "add52672f9e746059350930631bb2fe9"
format_stamp: "Formatted at 2026-09-26 13:45:00 on dist-test-slave-hc1k"
server_key: "a7a1897fb842a9b27cc4145d53aa5499"
server_key_iv: "a1b0f3047a190e70b2692d1b273876cb"
server_key_version: "encryptionkey@0"
I20260926 13:45:00.241372 15049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance:
uuid: "add52672f9e746059350930631bb2fe9"
format_stamp: "Formatted at 2026-09-26 13:45:00 on dist-test-slave-hc1k"
server_key: "a7a1897fb842a9b27cc4145d53aa5499"
server_key_iv: "a1b0f3047a190e70b2692d1b273876cb"
server_key_version: "encryptionkey@0"
I20260926 13:45:00.244794 15049 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:45:00.247030 15064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:00.248260 15049 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:45:00.248355 15049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "add52672f9e746059350930631bb2fe9"
format_stamp: "Formatted at 2026-09-26 13:45:00 on dist-test-slave-hc1k"
server_key: "a7a1897fb842a9b27cc4145d53aa5499"
server_key_iv: "a1b0f3047a190e70b2692d1b273876cb"
server_key_version: "encryptionkey@0"
I20260926 13:45:00.248440 15049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:00.273913 15049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:00.276495 15049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:00.276654 15049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:00.278219 15049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:00.278265 15049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:00.278314 15049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:00.278342 15049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 13:45:00 dist-test-slave-hc1k krb5kdc[14652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430300, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:00.288045 15049 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:38353
I20260926 13:45:00.288146 15179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:38353 every 8 connection(s)
I20260926 13:45:00.288971 15049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:45:00.291060 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 15049
I20260926 13:45:00.291180 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance
I20260926 13:45:00.291500 12969 external_mini_cluster.cc:1428] Setting key 8d8ba3559268839856ee3e7779807eb3
I20260926 13:45:00.291673 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.278952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:36613 (local address 127.12.170.126:46601)
0926 13:45:00.279115 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:00.279120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:00.279975 (+ 855us) server_negotiation.cc:407] Connection header received
0926 13:45:00.281203 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:00.281207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:00.281265 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:00.281351 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:00.282877 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.283522 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.285165 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.285440 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.285535 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.288394 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:00.288398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:00.288401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:00.288428 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:00.289849 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:00.290296 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:00.290298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:00.290299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:00.290363 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:00.290619 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:00.290621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:00.290622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:00.290773 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:00.290840 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:00.291407 (+ 567us) server_negotiation.cc:299] Negotiation successful
0926 13:45:00.291509 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 13:45:00.292541 15073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.279359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:46601 (local address 127.12.170.67:36613)
0926 13:45:00.279827 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:00.279878 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:00.280990 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:00.281593 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:00.281603 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:00.282043 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:00.282697 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.282718 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.283676 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.283680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.285046 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.285057 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.285604 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.285608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.285797 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.286342 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:00.286361 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:00.288236 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:00.289950 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:00.289954 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:00.289970 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:00.290216 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:00.290425 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:00.290427 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:00.290428 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:00.290547 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:00.290868 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:00.290875 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:00.291163 (+ 288us) client_negotiation.cc:769] Sending connection context
0926 13:45:00.291362 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 13:45:00.291697 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":105,"threads_started":1}
I20260926 13:45:00.293654 15180 heartbeater.cc:347] Connected to a master server at 127.12.170.126:46601
I20260926 13:45:00.293776 15180 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:00.294179 15180 heartbeater.cc:511] Master 127.12.170.126:46601 requested a full tablet report, sending...
I20260926 13:45:00.295007 14695 ts_manager.cc:194] Registered new tserver with Master: add52672f9e746059350930631bb2fe9 (127.12.170.67:38353)
I20260926 13:45:00.295527 14695 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.67@KRBTEST.COM'} at 127.12.170.67:36613
I20260926 13:45:00.301434 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.295525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:46601 (local address 127.12.170.67:47347)
0926 13:45:00.295847 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:00.295858 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:00.295929 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:00.296155 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:00.296157 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:00.296348 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:00.296448 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.296454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.297126 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.297130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.297705 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.297713 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.298096 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.298098 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.298161 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.298534 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:00.298545 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:00.298790 (+ 245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:00.300334 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:00.300337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:00.300338 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:00.300514 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:00.300761 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:00.300763 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:00.300764 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:00.300825 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:00.301108 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:00.301110 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:00.301171 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 13:45:00.301238 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 13:45:00.301323 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":83,"threads_started":1}
I20260926 13:45:00.301456 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.295602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:47347 (local address 127.12.170.126:46601)
0926 13:45:00.295720 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:00.295724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:00.295908 (+ 184us) server_negotiation.cc:407] Connection header received
0926 13:45:00.295990 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:00.295993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:00.296041 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:00.296118 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:00.296552 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.297014 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.297794 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.298002 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.298084 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.298897 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:00.298900 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:00.298902 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:00.298923 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:00.300248 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:00.300615 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:00.300618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:00.300619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:00.300691 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:00.300883 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:00.300885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:00.300886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:00.301022 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:00.301115 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:00.301282 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 13:45:00.301359 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 13:45:00.304665 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 13:45:00 dist-test-slave-hc1k krb5kdc[14652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 13:45:00 dist-test-slave-hc1k krb5kdc[14652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:00.326618 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.319637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38468 (local address 127.12.170.126:46601)
0926 13:45:00.319776 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:00.319780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:00.319793 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:45:00.319824 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:00.319826 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:00.319861 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:00.319934 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:00.320559 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.321050 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.321845 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.322054 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.322134 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.324219 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:00.324223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:00.324225 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:00.324248 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:00.325267 (+ 1019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:00.325616 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:00.325619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:00.325621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:00.325691 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:00.325910 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:00.325913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:00.325915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:00.326077 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:00.326186 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:00.326389 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 13:45:00.326483 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 13:45:00.329129 14695 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:45:00.332010 14695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:45:00.345698 15197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.339361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:43493 (local address 127.0.0.1:44608)
0926 13:45:00.340368 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:00.340415 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:00.340530 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:00.341211 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:00.341214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:00.341227 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:00.341341 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.341347 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.342744 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.342747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.344437 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.344448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.345222 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.345225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.345326 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.345363 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 13:45:00.345460 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 13:45:00.345569 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"mutex_wait_us":171,"thread_start_us":58,"threads_started":1}
I20260926 13:45:00.345757 15199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.339595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44608 (local address 127.12.170.66:43493)
0926 13:45:00.340546 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:00.340558 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:00.340581 (+ 23us) server_negotiation.cc:407] Connection header received
0926 13:45:00.340683 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:00.340689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:00.340953 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:00.341090 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:00.341449 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.342620 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.344576 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.345113 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.345206 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.345351 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:00.345495 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 13:45:00.345585 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":849,"thread_start_us":144,"threads_started":1}
I20260926 13:45:00.346120 15200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.339469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:41243 (local address 127.0.0.1:43268)
0926 13:45:00.340671 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:00.340683 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:00.340760 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:00.341154 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:00.341160 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:00.341180 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:00.341327 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.341334 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.342956 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.342959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.344437 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.344448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.345558 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.345561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.345721 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.345874 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 13:45:00.345938 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 13:45:00.345997 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1148,"mutex_wait_us":137,"thread_start_us":78,"threads_started":1}
I20260926 13:45:00.346510 15202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.340183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43268 (local address 127.12.170.65:41243)
0926 13:45:00.340535 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:00.340546 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:00.340760 (+ 214us) server_negotiation.cc:407] Connection header received
0926 13:45:00.340869 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:00.340873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:00.341021 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:00.341163 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:00.341451 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.342620 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.344556 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.345265 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.345347 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.345485 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:00.346279 (+ 794us) server_negotiation.cc:299] Negotiation successful
0926 13:45:00.346378 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":84,"threads_started":1}
I20260926 13:45:00.347270 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.339983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39642 (local address 127.12.170.67:38353)
0926 13:45:00.340357 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:00.340368 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:00.340807 (+ 439us) server_negotiation.cc:407] Connection header received
0926 13:45:00.341090 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:00.341095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:00.341247 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:00.341372 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:00.341575 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.342682 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.344473 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.345244 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.345347 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.345529 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:00.346341 (+ 812us) server_negotiation.cc:299] Negotiation successful
0926 13:45:00.346413 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":88,"threads_started":1}
I20260926 13:45:00.347533 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.339876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:38353 (local address 127.0.0.1:39642)
0926 13:45:00.340708 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:00.340721 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:00.340944 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:00.341367 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:00.341370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:00.341385 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:00.341483 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.341490 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.342807 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.342810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.344360 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.344369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.345574 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.345578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.345742 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.346045 (+ 303us) client_negotiation.cc:769] Sending connection context
0926 13:45:00.346110 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 13:45:00.346170 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":701,"thread_start_us":93,"threads_started":1}
I20260926 13:45:00.348589 14974 tablet_service.cc:1467] Processing CreateTablet for tablet ce948ad3baf5445fbf3ec4a5b1ab8546 (DEFAULT_TABLE table=test-table [id=d8c67e7ffc5f4843b44d310f5d4a6b30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:00.349151 14834 tablet_service.cc:1467] Processing CreateTablet for tablet ce948ad3baf5445fbf3ec4a5b1ab8546 (DEFAULT_TABLE table=test-table [id=d8c67e7ffc5f4843b44d310f5d4a6b30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:00.349655 14974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce948ad3baf5445fbf3ec4a5b1ab8546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:00.350121 14834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce948ad3baf5445fbf3ec4a5b1ab8546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:00.353768 15114 tablet_service.cc:1467] Processing CreateTablet for tablet ce948ad3baf5445fbf3ec4a5b1ab8546 (DEFAULT_TABLE table=test-table [id=d8c67e7ffc5f4843b44d310f5d4a6b30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:00.354655 15114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce948ad3baf5445fbf3ec4a5b1ab8546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:00.357333 15203 tablet_bootstrap.cc:492] T ce948ad3baf5445fbf3ec4a5b1ab8546 P a1299c2a5fc0416f8d90363abdb436e2: Bootstrap starting.
I20260926 13:45:00.358050 15204 tablet_bootstrap.cc:492] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016: Bootstrap starting.
I20260926 13:45:00.359591 15203 tablet_bootstrap.cc:654] T ce948ad3baf5445fbf3ec4a5b1ab8546 P a1299c2a5fc0416f8d90363abdb436e2: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:00.359642 15204 tablet_bootstrap.cc:654] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:00.360464 15203 log.cc:824] T ce948ad3baf5445fbf3ec4a5b1ab8546 P a1299c2a5fc0416f8d90363abdb436e2: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:00.360466 15204 log.cc:824] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:00.361112 15207 tablet_bootstrap.cc:492] T ce948ad3baf5445fbf3ec4a5b1ab8546 P add52672f9e746059350930631bb2fe9: Bootstrap starting.
I20260926 13:45:00.362226 15204 tablet_bootstrap.cc:492] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016: No bootstrap required, opened a new log
I20260926 13:45:00.362396 15204 ts_tablet_manager.cc:1397] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 13:45:00.362677 15207 tablet_bootstrap.cc:654] T ce948ad3baf5445fbf3ec4a5b1ab8546 P add52672f9e746059350930631bb2fe9: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:00.363246 15207 log.cc:824] T ce948ad3baf5445fbf3ec4a5b1ab8546 P add52672f9e746059350930631bb2fe9: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:00.364648 15203 tablet_bootstrap.cc:492] T ce948ad3baf5445fbf3ec4a5b1ab8546 P a1299c2a5fc0416f8d90363abdb436e2: No bootstrap required, opened a new log
I20260926 13:45:00.364809 15203 ts_tablet_manager.cc:1397] T ce948ad3baf5445fbf3ec4a5b1ab8546 P a1299c2a5fc0416f8d90363abdb436e2: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260926 13:45:00.365165 15207 tablet_bootstrap.cc:492] T ce948ad3baf5445fbf3ec4a5b1ab8546 P add52672f9e746059350930631bb2fe9: No bootstrap required, opened a new log
I20260926 13:45:00.365339 15207 ts_tablet_manager.cc:1397] T ce948ad3baf5445fbf3ec4a5b1ab8546 P add52672f9e746059350930631bb2fe9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 13:45:00.366005 15204 raft_consensus.cc:348] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:00.366273 15204 raft_consensus.cc:374] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:00.366333 15204 raft_consensus.cc:729] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41140ad3adc462ea3e3b92afebdf016, State: Initialized, Role: FOLLOWER
I20260926 13:45:00.366773 15204 consensus_queue.cc:260] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:00.367426 15040 heartbeater.cc:503] Master 127.12.170.126:46601 was elected leader, sending a full tablet report...
I20260926 13:45:00.367812 15204 ts_tablet_manager.cc:1428] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260926 13:45:00.367785 15203 raft_consensus.cc:348] T ce948ad3baf5445fbf3ec4a5b1ab8546 P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:00.367996 15203 raft_consensus.cc:374] T ce948ad3baf5445fbf3ec4a5b1ab8546 P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:00.368040 15203 raft_consensus.cc:729] T ce948ad3baf5445fbf3ec4a5b1ab8546 P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1299c2a5fc0416f8d90363abdb436e2, State: Initialized, Role: FOLLOWER
I20260926 13:45:00.368412 15203 consensus_queue.cc:260] T ce948ad3baf5445fbf3ec4a5b1ab8546 P a1299c2a5fc0416f8d90363abdb436e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:00.369019 14900 heartbeater.cc:503] Master 127.12.170.126:46601 was elected leader, sending a full tablet report...
I20260926 13:45:00.369299 15203 ts_tablet_manager.cc:1428] T ce948ad3baf5445fbf3ec4a5b1ab8546 P a1299c2a5fc0416f8d90363abdb436e2: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260926 13:45:00.370102 15207 raft_consensus.cc:348] T ce948ad3baf5445fbf3ec4a5b1ab8546 P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:00.370352 15207 raft_consensus.cc:374] T ce948ad3baf5445fbf3ec4a5b1ab8546 P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:00.370406 15207 raft_consensus.cc:729] T ce948ad3baf5445fbf3ec4a5b1ab8546 P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: add52672f9e746059350930631bb2fe9, State: Initialized, Role: FOLLOWER
I20260926 13:45:00.370777 15207 consensus_queue.cc:260] T ce948ad3baf5445fbf3ec4a5b1ab8546 P add52672f9e746059350930631bb2fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:00.371351 15180 heartbeater.cc:503] Master 127.12.170.126:46601 was elected leader, sending a full tablet report...
I20260926 13:45:00.371719 15207 ts_tablet_manager.cc:1428] T ce948ad3baf5445fbf3ec4a5b1ab8546 P add52672f9e746059350930631bb2fe9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260926 13:45:00.376364 14901 tablet.cc:2369] T ce948ad3baf5445fbf3ec4a5b1ab8546 P a1299c2a5fc0416f8d90363abdb436e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:45:00.539927 15181 tablet.cc:2369] T ce948ad3baf5445fbf3ec4a5b1ab8546 P add52672f9e746059350930631bb2fe9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:45:00.568401 15209 raft_consensus.cc:482] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:45:00.568593 15209 raft_consensus.cc:504] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:00.569681 15209 leader_election.cc:288] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1299c2a5fc0416f8d90363abdb436e2 (127.12.170.65:41243), add52672f9e746059350930631bb2fe9 (127.12.170.67:38353)
I20260926 13:45:00.573058 14933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.569921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:41243 (local address 127.12.170.66:60533)
0926 13:45:00.570084 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:00.570099 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:00.570196 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:00.570421 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:00.570424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:00.570439 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:00.570561 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.570566 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.571251 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.571254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.572068 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.572076 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.572660 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.572663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.572772 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.572788 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 13:45:00.572851 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 13:45:00.572920 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"spinlock_wait_cycles":27520}
I20260926 13:45:00.573047 15202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.570002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:60533 (local address 127.12.170.65:41243)
0926 13:45:00.570109 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:00.570114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:00.570151 (+ 37us) server_negotiation.cc:407] Connection header received
0926 13:45:00.570299 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:00.570302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:00.570342 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:00.570402 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:00.570662 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.571153 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.572181 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.572560 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.572658 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.572710 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:00.572876 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 13:45:00.572941 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 13:45:00.573707 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.570046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:46493 (local address 127.12.170.67:38353)
0926 13:45:00.570161 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:00.570165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:00.570454 (+ 289us) server_negotiation.cc:407] Connection header received
0926 13:45:00.570578 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:00.570581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:00.570623 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:00.570683 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:00.570927 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.571406 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.572811 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.573251 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.573361 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.573403 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:00.573562 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 13:45:00.573618 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 13:45:00.573664 14854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce948ad3baf5445fbf3ec4a5b1ab8546" candidate_uuid: "e41140ad3adc462ea3e3b92afebdf016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1299c2a5fc0416f8d90363abdb436e2" is_pre_election: true
I20260926 13:45:00.573745 15212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.569921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:38353 (local address 127.12.170.66:46493)
0926 13:45:00.570365 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:00.570375 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:00.570486 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:00.570704 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:00.570707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:00.570721 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:00.570833 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.570838 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.571509 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.571512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.572695 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.572704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.573355 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.573358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.573450 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.573472 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 13:45:00.573545 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 13:45:00.573606 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":178,"thread_start_us":74,"threads_started":1}
I20260926 13:45:00.573963 14854 raft_consensus.cc:2454] T ce948ad3baf5445fbf3ec4a5b1ab8546 P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e41140ad3adc462ea3e3b92afebdf016 in term 0.
I20260926 13:45:00.574292 15134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce948ad3baf5445fbf3ec4a5b1ab8546" candidate_uuid: "e41140ad3adc462ea3e3b92afebdf016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "add52672f9e746059350930631bb2fe9" is_pre_election: true
I20260926 13:45:00.574446 14925 leader_election.cc:302] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1299c2a5fc0416f8d90363abdb436e2, e41140ad3adc462ea3e3b92afebdf016; no voters:
I20260926 13:45:00.574532 15134 raft_consensus.cc:2454] T ce948ad3baf5445fbf3ec4a5b1ab8546 P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e41140ad3adc462ea3e3b92afebdf016 in term 0.
I20260926 13:45:00.574695 15209 raft_consensus.cc:2781] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:45:00.574784 15209 raft_consensus.cc:482] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:45:00.574836 15209 raft_consensus.cc:3037] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:00.575860 15209 raft_consensus.cc:504] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:00.576184 15209 leader_election.cc:288] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016 [CANDIDATE]: Term 1 election: Requested vote from peers a1299c2a5fc0416f8d90363abdb436e2 (127.12.170.65:41243), add52672f9e746059350930631bb2fe9 (127.12.170.67:38353)
I20260926 13:45:00.576438 14854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce948ad3baf5445fbf3ec4a5b1ab8546" candidate_uuid: "e41140ad3adc462ea3e3b92afebdf016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1299c2a5fc0416f8d90363abdb436e2"
I20260926 13:45:00.576476 15134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce948ad3baf5445fbf3ec4a5b1ab8546" candidate_uuid: "e41140ad3adc462ea3e3b92afebdf016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "add52672f9e746059350930631bb2fe9"
I20260926 13:45:00.576552 14854 raft_consensus.cc:3037] T ce948ad3baf5445fbf3ec4a5b1ab8546 P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:00.576572 15134 raft_consensus.cc:3037] T ce948ad3baf5445fbf3ec4a5b1ab8546 P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:00.577442 14854 raft_consensus.cc:2454] T ce948ad3baf5445fbf3ec4a5b1ab8546 P a1299c2a5fc0416f8d90363abdb436e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e41140ad3adc462ea3e3b92afebdf016 in term 1.
I20260926 13:45:00.577733 14925 leader_election.cc:302] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1299c2a5fc0416f8d90363abdb436e2, e41140ad3adc462ea3e3b92afebdf016; no voters:
I20260926 13:45:00.577895 15209 raft_consensus.cc:2781] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:00.577942 15134 raft_consensus.cc:2454] T ce948ad3baf5445fbf3ec4a5b1ab8546 P add52672f9e746059350930631bb2fe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e41140ad3adc462ea3e3b92afebdf016 in term 1.
I20260926 13:45:00.578120 15209 raft_consensus.cc:686] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016 [term 1 LEADER]: Becoming Leader. State: Replica: e41140ad3adc462ea3e3b92afebdf016, State: Running, Role: LEADER
I20260926 13:45:00.578419 15209 consensus_queue.cc:237] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
W20260926 13:45:00.580986 15041 tablet.cc:2369] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:45:00.581621 14697 catalog_manager.cc:5495] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016 reported cstate change: term changed from 0 to 1, leader changed from <none> to e41140ad3adc462ea3e3b92afebdf016 (127.12.170.66). New cstate: current_term: 1 leader_uuid: "e41140ad3adc462ea3e3b92afebdf016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } health_report { overall_health: UNKNOWN } } }
Sep 26 13:45:00 dist-test-slave-hc1k krb5kdc[14652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430299, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:00.656558 15220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.649060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:46601 (local address 127.0.0.1:38484)
0926 13:45:00.649318 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:00.649329 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:00.649429 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:00.649632 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:00.649635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:00.649999 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:00.650099 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.650106 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.650792 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.650795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.651341 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.651350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.651821 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.651824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.651899 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.652425 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:00.652439 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:00.653861 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:00.655168 (+ 1307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:00.655176 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:00.655182 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:00.655382 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:00.655609 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:00.655611 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:00.655613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:00.655712 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:00.656038 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:00.656044 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:00.656147 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 13:45:00.656237 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 13:45:00.656374 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":101,"threads_started":1}
I20260926 13:45:00.656558 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.649102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38484 (local address 127.12.170.126:46601)
0926 13:45:00.649203 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:00.649208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:00.649396 (+ 188us) server_negotiation.cc:407] Connection header received
0926 13:45:00.649506 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:00.649509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:00.649547 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:00.649610 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:00.650204 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.650699 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.651451 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.651721 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.651801 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.653974 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:00.653977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:00.653980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:00.654002 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:00.655078 (+ 1076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:00.655462 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:00.655465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:00.655467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:00.655537 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:00.655787 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:00.655790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:00.655791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:00.655950 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:00.656037 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:00.656290 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 13:45:00.656423 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":33}
I20260926 13:45:00.659906 14697 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:38484:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 13:45:00.660537 14697 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:45:00.664829 14834 tablet_service.cc:1467] Processing CreateTablet for tablet b165fcf2094f4b8cb9d92e9b6684333c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac85bee1adb74d1f9cfeabe31700f171]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:45:00.664956 14974 tablet_service.cc:1467] Processing CreateTablet for tablet b165fcf2094f4b8cb9d92e9b6684333c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac85bee1adb74d1f9cfeabe31700f171]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:45:00.665613 14834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b165fcf2094f4b8cb9d92e9b6684333c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:00.665647 14974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b165fcf2094f4b8cb9d92e9b6684333c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:00.664986 15114 tablet_service.cc:1467] Processing CreateTablet for tablet b165fcf2094f4b8cb9d92e9b6684333c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ac85bee1adb74d1f9cfeabe31700f171]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:45:00.666668 15114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b165fcf2094f4b8cb9d92e9b6684333c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:00.668152 15204 tablet_bootstrap.cc:492] T b165fcf2094f4b8cb9d92e9b6684333c P e41140ad3adc462ea3e3b92afebdf016: Bootstrap starting.
I20260926 13:45:00.668386 15203 tablet_bootstrap.cc:492] T b165fcf2094f4b8cb9d92e9b6684333c P a1299c2a5fc0416f8d90363abdb436e2: Bootstrap starting.
I20260926 13:45:00.668938 15204 tablet_bootstrap.cc:654] T b165fcf2094f4b8cb9d92e9b6684333c P e41140ad3adc462ea3e3b92afebdf016: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:00.669246 15203 tablet_bootstrap.cc:654] T b165fcf2094f4b8cb9d92e9b6684333c P a1299c2a5fc0416f8d90363abdb436e2: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:00.669327 15207 tablet_bootstrap.cc:492] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9: Bootstrap starting.
I20260926 13:45:00.670158 15207 tablet_bootstrap.cc:654] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:00.670341 15204 tablet_bootstrap.cc:492] T b165fcf2094f4b8cb9d92e9b6684333c P e41140ad3adc462ea3e3b92afebdf016: No bootstrap required, opened a new log
I20260926 13:45:00.670435 15204 ts_tablet_manager.cc:1397] T b165fcf2094f4b8cb9d92e9b6684333c P e41140ad3adc462ea3e3b92afebdf016: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:00.670804 15204 raft_consensus.cc:348] T b165fcf2094f4b8cb9d92e9b6684333c P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:00.670913 15204 raft_consensus.cc:374] T b165fcf2094f4b8cb9d92e9b6684333c P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:00.670945 15204 raft_consensus.cc:729] T b165fcf2094f4b8cb9d92e9b6684333c P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41140ad3adc462ea3e3b92afebdf016, State: Initialized, Role: FOLLOWER
I20260926 13:45:00.671064 15204 consensus_queue.cc:260] T b165fcf2094f4b8cb9d92e9b6684333c P e41140ad3adc462ea3e3b92afebdf016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:00.671257 15207 tablet_bootstrap.cc:492] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9: No bootstrap required, opened a new log
I20260926 13:45:00.671316 15204 ts_tablet_manager.cc:1428] T b165fcf2094f4b8cb9d92e9b6684333c P e41140ad3adc462ea3e3b92afebdf016: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 13:45:00.671330 15207 ts_tablet_manager.cc:1397] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:00.671279 15209 tablet_replica.cc:439] T b165fcf2094f4b8cb9d92e9b6684333c P e41140ad3adc462ea3e3b92afebdf016: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } } }
I20260926 13:45:00.671391 15209 tablet_replica.cc:442] T b165fcf2094f4b8cb9d92e9b6684333c P e41140ad3adc462ea3e3b92afebdf016: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:45:00.671779 15203 tablet_bootstrap.cc:492] T b165fcf2094f4b8cb9d92e9b6684333c P a1299c2a5fc0416f8d90363abdb436e2: No bootstrap required, opened a new log
I20260926 13:45:00.671743 15207 raft_consensus.cc:348] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:00.671819 15207 raft_consensus.cc:374] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:00.671845 15203 ts_tablet_manager.cc:1397] T b165fcf2094f4b8cb9d92e9b6684333c P a1299c2a5fc0416f8d90363abdb436e2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 13:45:00.671847 15207 raft_consensus.cc:729] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: add52672f9e746059350930631bb2fe9, State: Initialized, Role: FOLLOWER
I20260926 13:45:00.671957 15207 consensus_queue.cc:260] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:00.672189 15207 ts_tablet_manager.cc:1428] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:45:00.672231 15203 raft_consensus.cc:348] T b165fcf2094f4b8cb9d92e9b6684333c P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:00.672210 15211 tablet_replica.cc:439] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } } }
I20260926 13:45:00.672331 15203 raft_consensus.cc:374] T b165fcf2094f4b8cb9d92e9b6684333c P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:00.672349 15211 tablet_replica.cc:442] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:45:00.672369 15203 raft_consensus.cc:729] T b165fcf2094f4b8cb9d92e9b6684333c P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1299c2a5fc0416f8d90363abdb436e2, State: Initialized, Role: FOLLOWER
I20260926 13:45:00.672483 15203 consensus_queue.cc:260] T b165fcf2094f4b8cb9d92e9b6684333c P a1299c2a5fc0416f8d90363abdb436e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:00.672739 15203 ts_tablet_manager.cc:1428] T b165fcf2094f4b8cb9d92e9b6684333c P a1299c2a5fc0416f8d90363abdb436e2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:45:00.672914 15210 tablet_replica.cc:439] T b165fcf2094f4b8cb9d92e9b6684333c P a1299c2a5fc0416f8d90363abdb436e2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } } }
I20260926 13:45:00.673029 15210 tablet_replica.cc:442] T b165fcf2094f4b8cb9d92e9b6684333c P a1299c2a5fc0416f8d90363abdb436e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:45:00.840167 15211 raft_consensus.cc:482] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:45:00.840314 15211 raft_consensus.cc:504] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:00.841359 15211 leader_election.cc:288] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1299c2a5fc0416f8d90363abdb436e2 (127.12.170.65:41243), e41140ad3adc462ea3e3b92afebdf016 (127.12.170.66:43493)
I20260926 13:45:00.845206 15225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.841553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:41243 (local address 127.12.170.67:37167)
0926 13:45:00.841897 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:00.841911 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:00.842010 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:00.842287 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:00.842290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:00.842305 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:00.842432 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.842439 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.843215 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.843218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.844185 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.844194 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.844814 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.844817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.844907 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.844931 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 13:45:00.844993 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 13:45:00.845055 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":44,"thread_start_us":68,"threads_started":1}
I20260926 13:45:00.845232 15202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.841900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:37167 (local address 127.12.170.65:41243)
0926 13:45:00.842052 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:00.842057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:00.842075 (+ 18us) server_negotiation.cc:407] Connection header received
0926 13:45:00.842138 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:00.842141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:00.842190 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:00.842281 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:00.842535 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.843112 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.844297 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.844706 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.844813 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.844872 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:00.845021 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 13:45:00.845093 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 13:45:00.845307 15199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.841742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:40083 (local address 127.12.170.66:43493)
0926 13:45:00.841877 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:00.841882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:00.842145 (+ 263us) server_negotiation.cc:407] Connection header received
0926 13:45:00.842255 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:00.842258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:00.842305 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:00.842376 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:00.842657 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.843133 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.844335 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.844735 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.844842 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.844891 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:00.845129 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 13:45:00.845204 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 13:45:00.845769 15224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.841647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:43493 (local address 127.12.170.67:40083)
0926 13:45:00.842043 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:00.842059 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:00.842168 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:00.842402 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:00.842405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:00.842422 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:00.842553 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.842561 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.843237 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.843240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.844211 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.844221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.844843 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.844848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.844965 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.844990 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 13:45:00.845109 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 13:45:00.845177 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":67,"thread_start_us":83,"threads_started":1}
I20260926 13:45:00.845767 14854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b165fcf2094f4b8cb9d92e9b6684333c" candidate_uuid: "add52672f9e746059350930631bb2fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1299c2a5fc0416f8d90363abdb436e2" is_pre_election: true
I20260926 13:45:00.845899 14854 raft_consensus.cc:2454] T b165fcf2094f4b8cb9d92e9b6684333c P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate add52672f9e746059350930631bb2fe9 in term 0.
I20260926 13:45:00.846228 15065 leader_election.cc:302] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1299c2a5fc0416f8d90363abdb436e2, add52672f9e746059350930631bb2fe9; no voters:
I20260926 13:45:00.846385 14994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b165fcf2094f4b8cb9d92e9b6684333c" candidate_uuid: "add52672f9e746059350930631bb2fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41140ad3adc462ea3e3b92afebdf016" is_pre_election: true
I20260926 13:45:00.846522 15211 raft_consensus.cc:2781] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:45:00.846613 15211 raft_consensus.cc:482] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:45:00.846633 14994 raft_consensus.cc:2454] T b165fcf2094f4b8cb9d92e9b6684333c P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate add52672f9e746059350930631bb2fe9 in term 0.
I20260926 13:45:00.846664 15211 raft_consensus.cc:3037] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:00.847918 15211 raft_consensus.cc:504] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:00.848318 15211 leader_election.cc:288] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9 [CANDIDATE]: Term 1 election: Requested vote from peers a1299c2a5fc0416f8d90363abdb436e2 (127.12.170.65:41243), e41140ad3adc462ea3e3b92afebdf016 (127.12.170.66:43493)
I20260926 13:45:00.848578 14854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b165fcf2094f4b8cb9d92e9b6684333c" candidate_uuid: "add52672f9e746059350930631bb2fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1299c2a5fc0416f8d90363abdb436e2"
I20260926 13:45:00.848696 14854 raft_consensus.cc:3037] T b165fcf2094f4b8cb9d92e9b6684333c P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:00.848778 14994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b165fcf2094f4b8cb9d92e9b6684333c" candidate_uuid: "add52672f9e746059350930631bb2fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41140ad3adc462ea3e3b92afebdf016"
I20260926 13:45:00.848875 14994 raft_consensus.cc:3037] T b165fcf2094f4b8cb9d92e9b6684333c P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:00.849737 14854 raft_consensus.cc:2454] T b165fcf2094f4b8cb9d92e9b6684333c P a1299c2a5fc0416f8d90363abdb436e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate add52672f9e746059350930631bb2fe9 in term 1.
I20260926 13:45:00.849905 14994 raft_consensus.cc:2454] T b165fcf2094f4b8cb9d92e9b6684333c P e41140ad3adc462ea3e3b92afebdf016 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate add52672f9e746059350930631bb2fe9 in term 1.
I20260926 13:45:00.849995 15065 leader_election.cc:302] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1299c2a5fc0416f8d90363abdb436e2, add52672f9e746059350930631bb2fe9; no voters:
I20260926 13:45:00.850147 15211 raft_consensus.cc:2781] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:00.850325 15211 raft_consensus.cc:686] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9 [term 1 LEADER]: Becoming Leader. State: Replica: add52672f9e746059350930631bb2fe9, State: Running, Role: LEADER
I20260926 13:45:00.850704 15211 consensus_queue.cc:237] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:00.852927 15226 tablet_replica.cc:439] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9: TxnStatusTablet state changed. Reason: New leader add52672f9e746059350930631bb2fe9. Latest consensus state: current_term: 1 leader_uuid: "add52672f9e746059350930631bb2fe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } } }
I20260926 13:45:00.853093 15226 tablet_replica.cc:442] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:45:00.853363 15228 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 13:45:00.853626 15228 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 13:45:00.854131 14695 catalog_manager.cc:5495] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9 reported cstate change: term changed from 0 to 1, leader changed from <none> to add52672f9e746059350930631bb2fe9 (127.12.170.67). New cstate: current_term: 1 leader_uuid: "add52672f9e746059350930631bb2fe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } health_report { overall_health: HEALTHY } } }
I20260926 13:45:00.903525 15220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.899917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:38353 (local address 127.0.0.1:39658)
0926 13:45:00.900166 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:00.900181 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:00.900291 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:00.900604 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:00.900607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:00.900614 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:45:00.900718 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.900724 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.901415 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.901418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.902037 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.902045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.902456 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.902459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.902550 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.902603 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:45:00.903248 (+ 645us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:45:00.903264 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 13:45:00.903325 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 13:45:00.903411 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260926 13:45:00.903525 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.900009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39658 (local address 127.12.170.67:38353)
0926 13:45:00.900148 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:00.900152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:00.900237 (+ 85us) server_negotiation.cc:407] Connection header received
0926 13:45:00.900430 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:00.900433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:00.900486 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:00.900567 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:45:00.900814 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.901309 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.902149 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.902358 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.902443 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.902736 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:45:00.903153 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:45:00.903250 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:00.903335 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 13:45:00.903387 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 13:45:00.908370 14994 raft_consensus.cc:1261] T b165fcf2094f4b8cb9d92e9b6684333c P e41140ad3adc462ea3e3b92afebdf016 [term 1 FOLLOWER]: Refusing update from remote peer add52672f9e746059350930631bb2fe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:00.908526 14854 raft_consensus.cc:1261] T b165fcf2094f4b8cb9d92e9b6684333c P a1299c2a5fc0416f8d90363abdb436e2 [term 1 FOLLOWER]: Refusing update from remote peer add52672f9e746059350930631bb2fe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:00.908958 15226 consensus_queue.cc:1035] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:00.909116 15211 consensus_queue.cc:1035] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:00.912321 15209 tablet_replica.cc:439] T b165fcf2094f4b8cb9d92e9b6684333c P e41140ad3adc462ea3e3b92afebdf016: TxnStatusTablet state changed. Reason: New leader add52672f9e746059350930631bb2fe9. Latest consensus state: current_term: 1 leader_uuid: "add52672f9e746059350930631bb2fe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } } }
I20260926 13:45:00.912454 15209 tablet_replica.cc:442] T b165fcf2094f4b8cb9d92e9b6684333c P e41140ad3adc462ea3e3b92afebdf016: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:45:00.913028 15210 tablet_replica.cc:439] T b165fcf2094f4b8cb9d92e9b6684333c P a1299c2a5fc0416f8d90363abdb436e2: TxnStatusTablet state changed. Reason: New leader add52672f9e746059350930631bb2fe9. Latest consensus state: current_term: 1 leader_uuid: "add52672f9e746059350930631bb2fe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } } }
I20260926 13:45:00.913146 15210 tablet_replica.cc:442] T b165fcf2094f4b8cb9d92e9b6684333c P a1299c2a5fc0416f8d90363abdb436e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:45:00.916160 15211 tablet_replica.cc:439] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "add52672f9e746059350930631bb2fe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } } }
I20260926 13:45:00.916270 15211 tablet_replica.cc:442] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:45:00.916380 15211 tablet_replica.cc:439] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "add52672f9e746059350930631bb2fe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } } }
I20260926 13:45:00.916450 15211 tablet_replica.cc:442] T b165fcf2094f4b8cb9d92e9b6684333c P add52672f9e746059350930631bb2fe9: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:45:00.916532 15231 mvcc.cc:205] Tried to move back new op lower bound from 7333602512514707456 to 7333602512287178752. Current Snapshot: MvccSnapshot[applied={T|T < 7333602512514707456}]
I20260926 13:45:00.916970 15210 tablet_replica.cc:439] T b165fcf2094f4b8cb9d92e9b6684333c P a1299c2a5fc0416f8d90363abdb436e2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "add52672f9e746059350930631bb2fe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } } }
I20260926 13:45:00.917065 15210 tablet_replica.cc:442] T b165fcf2094f4b8cb9d92e9b6684333c P a1299c2a5fc0416f8d90363abdb436e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:45:00.917176 15209 tablet_replica.cc:439] T b165fcf2094f4b8cb9d92e9b6684333c P e41140ad3adc462ea3e3b92afebdf016: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "add52672f9e746059350930631bb2fe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } } }
I20260926 13:45:00.917268 15209 tablet_replica.cc:442] T b165fcf2094f4b8cb9d92e9b6684333c P e41140ad3adc462ea3e3b92afebdf016: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:45:00.917366 15232 mvcc.cc:205] Tried to move back new op lower bound from 7333602512514707456 to 7333602512287178752. Current Snapshot: MvccSnapshot[applied={T|T < 7333602512514707456}]
I20260926 13:45:00.920282 15233 mvcc.cc:205] Tried to move back new op lower bound from 7333602512514707456 to 7333602512287178752. Current Snapshot: MvccSnapshot[applied={T|T < 7333602512514707456}]
I20260926 13:45:00.926998 15199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.923292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44618 (local address 127.12.170.66:43493)
0926 13:45:00.923411 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:00.923415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:00.923621 (+ 206us) server_negotiation.cc:407] Connection header received
0926 13:45:00.923785 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:00.923787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:00.923836 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:00.923905 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:45:00.924136 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.924622 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.925815 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.926016 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.926077 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.926333 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:45:00.926687 (+ 354us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:45:00.926762 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:00.926847 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 13:45:00.926890 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:45:00.936930 15243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.933592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:38353 (local address 127.12.170.66:45351)
0926 13:45:00.933915 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:00.933927 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:00.934009 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:00.934211 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:00.934213 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:00.934218 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:45:00.934305 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.934309 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.935214 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.935216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.935835 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.935841 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.936355 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.936357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.936418 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.936453 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:45:00.936711 (+ 258us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:45:00.936714 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:45:00.936754 (+ 40us) client_negotiation.cc:240] Negotiation successful
0926 13:45:00.936810 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":93,"threads_started":1}
I20260926 13:45:00.936932 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.933676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:45351 (local address 127.12.170.67:38353)
0926 13:45:00.933777 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:00.933781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:00.934001 (+ 220us) server_negotiation.cc:407] Connection header received
0926 13:45:00.934076 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:00.934079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:00.934128 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:00.934195 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:45:00.934383 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.935133 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.935928 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.936266 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.936364 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.936575 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:45:00.936645 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:45:00.936711 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:00.936766 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 13:45:00.936813 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 13:45:00.942602 14854 raft_consensus.cc:1261] T ce948ad3baf5445fbf3ec4a5b1ab8546 P a1299c2a5fc0416f8d90363abdb436e2 [term 1 FOLLOWER]: Refusing update from remote peer e41140ad3adc462ea3e3b92afebdf016: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:00.942790 15134 raft_consensus.cc:1261] T ce948ad3baf5445fbf3ec4a5b1ab8546 P add52672f9e746059350930631bb2fe9 [term 1 FOLLOWER]: Refusing update from remote peer e41140ad3adc462ea3e3b92afebdf016: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:00.943082 15213 consensus_queue.cc:1035] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:00.943240 15209 consensus_queue.cc:1035] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:00.965284 15199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.961875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:47685 (local address 127.12.170.66:43493)
0926 13:45:00.961976 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:00.961980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:00.962168 (+ 188us) server_negotiation.cc:407] Connection header received
0926 13:45:00.962284 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:00.962287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:00.962333 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:00.962385 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:45:00.962630 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.963191 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.964292 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.964508 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.964590 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.964884 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:45:00.964953 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:45:00.965008 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:00.965102 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 13:45:00.965153 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 13:45:00.965306 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:00.961815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:43493 (local address 127.12.170.67:47685)
0926 13:45:00.962095 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:00.962106 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:00.962199 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:00.962400 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:00.962402 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:00.962409 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:45:00.962536 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.962541 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.963323 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.963326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.964183 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:00.964193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:00.964627 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:00.964631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:00.964726 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:00.964765 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:45:00.965020 (+ 255us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:45:00.965027 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 13:45:00.965085 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 13:45:00.965157 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":95,"threads_started":1}
I20260926 13:45:00.992341 14697 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:45:00.992515 14697 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:45:00.995005 14697 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4: Sending DeleteTablet for 3 replicas of tablet ce948ad3baf5445fbf3ec4a5b1ab8546
I20260926 13:45:00.995680 14834 tablet_service.cc:1514] Processing DeleteTablet for tablet ce948ad3baf5445fbf3ec4a5b1ab8546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:00 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:43268
I20260926 13:45:00.995743 14974 tablet_service.cc:1514] Processing DeleteTablet for tablet ce948ad3baf5445fbf3ec4a5b1ab8546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:00 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:44608
I20260926 13:45:00.996371 15254 tablet_replica.cc:330] T ce948ad3baf5445fbf3ec4a5b1ab8546 P a1299c2a5fc0416f8d90363abdb436e2: stopping tablet replica
I20260926 13:45:00.996384 15114 tablet_service.cc:1514] Processing DeleteTablet for tablet ce948ad3baf5445fbf3ec4a5b1ab8546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:00 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:39642
I20260926 13:45:00.996913 15256 tablet_replica.cc:330] T ce948ad3baf5445fbf3ec4a5b1ab8546 P add52672f9e746059350930631bb2fe9: stopping tablet replica
I20260926 13:45:00.997156 15256 raft_consensus.cc:2229] T ce948ad3baf5445fbf3ec4a5b1ab8546 P add52672f9e746059350930631bb2fe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:45:00.997288 15256 raft_consensus.cc:2258] T ce948ad3baf5445fbf3ec4a5b1ab8546 P add52672f9e746059350930631bb2fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:00.997464 14697 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:45:00.997824 14697 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:45:00.998368 15256 ts_tablet_manager.cc:1905] T ce948ad3baf5445fbf3ec4a5b1ab8546 P add52672f9e746059350930631bb2fe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:00.996773 15254 raft_consensus.cc:2229] T ce948ad3baf5445fbf3ec4a5b1ab8546 P a1299c2a5fc0416f8d90363abdb436e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:45:00.999296 15254 raft_consensus.cc:2258] T ce948ad3baf5445fbf3ec4a5b1ab8546 P a1299c2a5fc0416f8d90363abdb436e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:00.999720 15255 tablet_replica.cc:330] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016: stopping tablet replica
I20260926 13:45:01.000013 15255 raft_consensus.cc:2229] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:45:01.000385 15255 raft_consensus.cc:2258] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:01.000559 15254 ts_tablet_manager.cc:1905] T ce948ad3baf5445fbf3ec4a5b1ab8546 P a1299c2a5fc0416f8d90363abdb436e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:01.001513 15255 ts_tablet_manager.cc:1905] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:01.002641 14834 tablet_service.cc:1467] Processing CreateTablet for tablet 8bce5014ff454322be85812a6d5196c2 (DEFAULT_TABLE table=test-table [id=48928b185b5249b9b2acbdada1326caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:01.002878 14834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bce5014ff454322be85812a6d5196c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:01.003387 15254 ts_tablet_manager.cc:1918] T ce948ad3baf5445fbf3ec4a5b1ab8546 P a1299c2a5fc0416f8d90363abdb436e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:45:01.003481 15254 log.cc:1196] T ce948ad3baf5445fbf3ec4a5b1ab8546 P a1299c2a5fc0416f8d90363abdb436e2: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-0/wal/wals/ce948ad3baf5445fbf3ec4a5b1ab8546
I20260926 13:45:01.003804 15254 ts_tablet_manager.cc:1939] T ce948ad3baf5445fbf3ec4a5b1ab8546 P a1299c2a5fc0416f8d90363abdb436e2: Deleting consensus metadata
I20260926 13:45:01.004637 15255 ts_tablet_manager.cc:1918] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:45:01.004787 15255 log.cc:1196] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-1/wal/wals/ce948ad3baf5445fbf3ec4a5b1ab8546
I20260926 13:45:01.004947 14681 catalog_manager.cc:4841] TS a1299c2a5fc0416f8d90363abdb436e2 (127.12.170.65:41243): tablet ce948ad3baf5445fbf3ec4a5b1ab8546 (table test-table [id=d8c67e7ffc5f4843b44d310f5d4a6b30]) successfully deleted
I20260926 13:45:01.005239 15255 ts_tablet_manager.cc:1939] T ce948ad3baf5445fbf3ec4a5b1ab8546 P e41140ad3adc462ea3e3b92afebdf016: Deleting consensus metadata
I20260926 13:45:01.005194 14974 tablet_service.cc:1467] Processing CreateTablet for tablet 8bce5014ff454322be85812a6d5196c2 (DEFAULT_TABLE table=test-table [id=48928b185b5249b9b2acbdada1326caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:01.005474 14974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bce5014ff454322be85812a6d5196c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:01.005504 15114 tablet_service.cc:1467] Processing CreateTablet for tablet 8bce5014ff454322be85812a6d5196c2 (DEFAULT_TABLE table=test-table [id=48928b185b5249b9b2acbdada1326caf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:01.005834 15203 tablet_bootstrap.cc:492] T 8bce5014ff454322be85812a6d5196c2 P a1299c2a5fc0416f8d90363abdb436e2: Bootstrap starting.
I20260926 13:45:01.006707 15203 tablet_bootstrap.cc:654] T 8bce5014ff454322be85812a6d5196c2 P a1299c2a5fc0416f8d90363abdb436e2: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:01.007481 15204 tablet_bootstrap.cc:492] T 8bce5014ff454322be85812a6d5196c2 P e41140ad3adc462ea3e3b92afebdf016: Bootstrap starting.
I20260926 13:45:01.008522 15204 tablet_bootstrap.cc:654] T 8bce5014ff454322be85812a6d5196c2 P e41140ad3adc462ea3e3b92afebdf016: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:01.008543 14683 catalog_manager.cc:4841] TS e41140ad3adc462ea3e3b92afebdf016 (127.12.170.66:43493): tablet ce948ad3baf5445fbf3ec4a5b1ab8546 (table test-table [id=d8c67e7ffc5f4843b44d310f5d4a6b30]) successfully deleted
I20260926 13:45:01.009136 15114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bce5014ff454322be85812a6d5196c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:01.011307 15207 tablet_bootstrap.cc:492] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9: Bootstrap starting.
I20260926 13:45:01.012571 15207 tablet_bootstrap.cc:654] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:01.013468 15256 ts_tablet_manager.cc:1918] T ce948ad3baf5445fbf3ec4a5b1ab8546 P add52672f9e746059350930631bb2fe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:45:01.013603 15256 log.cc:1196] T ce948ad3baf5445fbf3ec4a5b1ab8546 P add52672f9e746059350930631bb2fe9: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/ce948ad3baf5445fbf3ec4a5b1ab8546
I20260926 13:45:01.013950 15207 tablet_bootstrap.cc:492] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9: No bootstrap required, opened a new log
I20260926 13:45:01.014030 15207 ts_tablet_manager.cc:1397] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 13:45:01.014281 15256 ts_tablet_manager.cc:1939] T ce948ad3baf5445fbf3ec4a5b1ab8546 P add52672f9e746059350930631bb2fe9: Deleting consensus metadata
I20260926 13:45:01.014390 15207 raft_consensus.cc:348] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:01.014508 15207 raft_consensus.cc:374] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:01.014539 15207 raft_consensus.cc:729] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: add52672f9e746059350930631bb2fe9, State: Initialized, Role: FOLLOWER
I20260926 13:45:01.014638 15207 consensus_queue.cc:260] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:01.014851 15207 ts_tablet_manager.cc:1428] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:45:01.015285 15203 tablet_bootstrap.cc:492] T 8bce5014ff454322be85812a6d5196c2 P a1299c2a5fc0416f8d90363abdb436e2: No bootstrap required, opened a new log
I20260926 13:45:01.015362 15203 ts_tablet_manager.cc:1397] T 8bce5014ff454322be85812a6d5196c2 P a1299c2a5fc0416f8d90363abdb436e2: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260926 13:45:01.015760 15203 raft_consensus.cc:348] T 8bce5014ff454322be85812a6d5196c2 P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:01.015857 15203 raft_consensus.cc:374] T 8bce5014ff454322be85812a6d5196c2 P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:01.015882 15203 raft_consensus.cc:729] T 8bce5014ff454322be85812a6d5196c2 P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1299c2a5fc0416f8d90363abdb436e2, State: Initialized, Role: FOLLOWER
I20260926 13:45:01.015975 15203 consensus_queue.cc:260] T 8bce5014ff454322be85812a6d5196c2 P a1299c2a5fc0416f8d90363abdb436e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:01.016119 15204 tablet_bootstrap.cc:492] T 8bce5014ff454322be85812a6d5196c2 P e41140ad3adc462ea3e3b92afebdf016: No bootstrap required, opened a new log
I20260926 13:45:01.016186 15203 ts_tablet_manager.cc:1428] T 8bce5014ff454322be85812a6d5196c2 P a1299c2a5fc0416f8d90363abdb436e2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:45:01.016196 15204 ts_tablet_manager.cc:1397] T 8bce5014ff454322be85812a6d5196c2 P e41140ad3adc462ea3e3b92afebdf016: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260926 13:45:01.016583 15204 raft_consensus.cc:348] T 8bce5014ff454322be85812a6d5196c2 P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:01.016674 15204 raft_consensus.cc:374] T 8bce5014ff454322be85812a6d5196c2 P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:01.016701 15204 raft_consensus.cc:729] T 8bce5014ff454322be85812a6d5196c2 P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41140ad3adc462ea3e3b92afebdf016, State: Initialized, Role: FOLLOWER
I20260926 13:45:01.016805 15204 consensus_queue.cc:260] T 8bce5014ff454322be85812a6d5196c2 P e41140ad3adc462ea3e3b92afebdf016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:01.017030 15204 ts_tablet_manager.cc:1428] T 8bce5014ff454322be85812a6d5196c2 P e41140ad3adc462ea3e3b92afebdf016: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:45:01.017287 14682 catalog_manager.cc:4841] TS add52672f9e746059350930631bb2fe9 (127.12.170.67:38353): tablet ce948ad3baf5445fbf3ec4a5b1ab8546 (table test-table [id=d8c67e7ffc5f4843b44d310f5d4a6b30]) successfully deleted
I20260926 13:45:01.102837 15252 raft_consensus.cc:482] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:45:01.102974 15252 raft_consensus.cc:504] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:01.103432 15252 leader_election.cc:288] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1299c2a5fc0416f8d90363abdb436e2 (127.12.170.65:41243), e41140ad3adc462ea3e3b92afebdf016 (127.12.170.66:43493)
I20260926 13:45:01.103881 14994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bce5014ff454322be85812a6d5196c2" candidate_uuid: "add52672f9e746059350930631bb2fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41140ad3adc462ea3e3b92afebdf016" is_pre_election: true
I20260926 13:45:01.103932 14854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bce5014ff454322be85812a6d5196c2" candidate_uuid: "add52672f9e746059350930631bb2fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1299c2a5fc0416f8d90363abdb436e2" is_pre_election: true
I20260926 13:45:01.104032 14994 raft_consensus.cc:2454] T 8bce5014ff454322be85812a6d5196c2 P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate add52672f9e746059350930631bb2fe9 in term 0.
I20260926 13:45:01.104104 14854 raft_consensus.cc:2454] T 8bce5014ff454322be85812a6d5196c2 P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate add52672f9e746059350930631bb2fe9 in term 0.
I20260926 13:45:01.104316 15067 leader_election.cc:302] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: add52672f9e746059350930631bb2fe9, e41140ad3adc462ea3e3b92afebdf016; no voters:
I20260926 13:45:01.104528 15252 raft_consensus.cc:2781] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:45:01.104600 15252 raft_consensus.cc:482] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:45:01.104648 15252 raft_consensus.cc:3037] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:01.106001 15252 raft_consensus.cc:504] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:01.106361 15252 leader_election.cc:288] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9 [CANDIDATE]: Term 1 election: Requested vote from peers a1299c2a5fc0416f8d90363abdb436e2 (127.12.170.65:41243), e41140ad3adc462ea3e3b92afebdf016 (127.12.170.66:43493)
I20260926 13:45:01.106632 14854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bce5014ff454322be85812a6d5196c2" candidate_uuid: "add52672f9e746059350930631bb2fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1299c2a5fc0416f8d90363abdb436e2"
I20260926 13:45:01.106756 14854 raft_consensus.cc:3037] T 8bce5014ff454322be85812a6d5196c2 P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:01.106751 14994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bce5014ff454322be85812a6d5196c2" candidate_uuid: "add52672f9e746059350930631bb2fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41140ad3adc462ea3e3b92afebdf016"
I20260926 13:45:01.106873 14994 raft_consensus.cc:3037] T 8bce5014ff454322be85812a6d5196c2 P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:01.107921 14994 raft_consensus.cc:2454] T 8bce5014ff454322be85812a6d5196c2 P e41140ad3adc462ea3e3b92afebdf016 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate add52672f9e746059350930631bb2fe9 in term 1.
I20260926 13:45:01.107921 14854 raft_consensus.cc:2454] T 8bce5014ff454322be85812a6d5196c2 P a1299c2a5fc0416f8d90363abdb436e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate add52672f9e746059350930631bb2fe9 in term 1.
I20260926 13:45:01.108237 15067 leader_election.cc:302] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: add52672f9e746059350930631bb2fe9, e41140ad3adc462ea3e3b92afebdf016; no voters:
I20260926 13:45:01.108436 15252 raft_consensus.cc:2781] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:01.108552 15252 raft_consensus.cc:686] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9 [term 1 LEADER]: Becoming Leader. State: Replica: add52672f9e746059350930631bb2fe9, State: Running, Role: LEADER
I20260926 13:45:01.108696 15252 consensus_queue.cc:237] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:01.110395 14695 catalog_manager.cc:5495] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9 reported cstate change: term changed from 0 to 1, leader changed from <none> to add52672f9e746059350930631bb2fe9 (127.12.170.67). New cstate: current_term: 1 leader_uuid: "add52672f9e746059350930631bb2fe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } health_report { overall_health: HEALTHY } } }
I20260926 13:45:01.145882 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:01.142945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39674 (local address 127.12.170.67:38353)
0926 13:45:01.143069 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:01.143073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:01.143088 (+ 15us) server_negotiation.cc:407] Connection header received
0926 13:45:01.143231 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:01.143234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:01.143280 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:01.143337 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:45:01.143559 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:01.144106 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:01.144899 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:01.145121 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:01.145203 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:01.145453 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:45:01.145519 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:45:01.145584 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:01.145683 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 13:45:01.145734 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 13:45:01.154104 15201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:01.151361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:37609 (local address 127.12.170.67:38353)
0926 13:45:01.151459 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:01.151462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:01.151474 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:45:01.151579 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:01.151581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:01.151617 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:01.151693 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:45:01.151864 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:01.152378 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:01.153221 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:01.153483 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:01.153560 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:01.153754 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:45:01.153821 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:45:01.153865 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:01.153928 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 13:45:01.153961 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 13:45:01.154117 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:01.151299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:38353 (local address 127.12.170.67:37609)
0926 13:45:01.151426 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:01.151438 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:01.151503 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:01.151712 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:01.151714 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:01.151719 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:45:01.151802 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:01.151807 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:01.152455 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:01.152459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:01.153102 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:01.153111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:01.153579 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:01.153582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:01.153641 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:01.153665 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:45:01.153871 (+ 206us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:45:01.153876 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:45:01.153921 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 13:45:01.153979 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 13:45:01.159310 14854 raft_consensus.cc:1261] T 8bce5014ff454322be85812a6d5196c2 P a1299c2a5fc0416f8d90363abdb436e2 [term 1 FOLLOWER]: Refusing update from remote peer add52672f9e746059350930631bb2fe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:01.159336 14994 raft_consensus.cc:1261] T 8bce5014ff454322be85812a6d5196c2 P e41140ad3adc462ea3e3b92afebdf016 [term 1 FOLLOWER]: Refusing update from remote peer add52672f9e746059350930631bb2fe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:01.159644 15211 consensus_queue.cc:1035] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:01.159859 15226 consensus_queue.cc:1035] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 13:45:01.171415 15231 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 13:45:02.718964 14695 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:45:02.719105 14695 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:45:02.721127 14695 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4: Sending DeleteTablet for 3 replicas of tablet 8bce5014ff454322be85812a6d5196c2
I20260926 13:45:02.721591 14834 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bce5014ff454322be85812a6d5196c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:02 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:43268
I20260926 13:45:02.721586 14974 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bce5014ff454322be85812a6d5196c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:02 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:44608
I20260926 13:45:02.721721 15113 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bce5014ff454322be85812a6d5196c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:02 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:39642
I20260926 13:45:02.721900 15284 tablet_replica.cc:330] T 8bce5014ff454322be85812a6d5196c2 P a1299c2a5fc0416f8d90363abdb436e2: stopping tablet replica
I20260926 13:45:02.721997 15286 tablet_replica.cc:330] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9: stopping tablet replica
I20260926 13:45:02.722012 15284 raft_consensus.cc:2229] T 8bce5014ff454322be85812a6d5196c2 P a1299c2a5fc0416f8d90363abdb436e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:45:02.722105 15284 raft_consensus.cc:2258] T 8bce5014ff454322be85812a6d5196c2 P a1299c2a5fc0416f8d90363abdb436e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:02.722100 15286 raft_consensus.cc:2229] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:45:02.722348 15286 raft_consensus.cc:2258] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:02.722661 15284 ts_tablet_manager.cc:1905] T 8bce5014ff454322be85812a6d5196c2 P a1299c2a5fc0416f8d90363abdb436e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:02.722841 15286 ts_tablet_manager.cc:1905] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:02.724910 15285 tablet_replica.cc:330] T 8bce5014ff454322be85812a6d5196c2 P e41140ad3adc462ea3e3b92afebdf016: stopping tablet replica
I20260926 13:45:02.725062 15285 raft_consensus.cc:2229] T 8bce5014ff454322be85812a6d5196c2 P e41140ad3adc462ea3e3b92afebdf016 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:45:02.725128 15284 ts_tablet_manager.cc:1918] T 8bce5014ff454322be85812a6d5196c2 P a1299c2a5fc0416f8d90363abdb436e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 13:45:02.725206 15284 log.cc:1196] T 8bce5014ff454322be85812a6d5196c2 P a1299c2a5fc0416f8d90363abdb436e2: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-0/wal/wals/8bce5014ff454322be85812a6d5196c2
I20260926 13:45:02.725253 15286 ts_tablet_manager.cc:1918] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 13:45:02.725231 15285 raft_consensus.cc:2258] T 8bce5014ff454322be85812a6d5196c2 P e41140ad3adc462ea3e3b92afebdf016 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:02.725322 15286 log.cc:1196] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/8bce5014ff454322be85812a6d5196c2
I20260926 13:45:02.725452 15284 ts_tablet_manager.cc:1939] T 8bce5014ff454322be85812a6d5196c2 P a1299c2a5fc0416f8d90363abdb436e2: Deleting consensus metadata
I20260926 13:45:02.725550 15286 ts_tablet_manager.cc:1939] T 8bce5014ff454322be85812a6d5196c2 P add52672f9e746059350930631bb2fe9: Deleting consensus metadata
I20260926 13:45:02.725919 15285 ts_tablet_manager.cc:1905] T 8bce5014ff454322be85812a6d5196c2 P e41140ad3adc462ea3e3b92afebdf016: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:02.726053 14682 catalog_manager.cc:4841] TS add52672f9e746059350930631bb2fe9 (127.12.170.67:38353): tablet 8bce5014ff454322be85812a6d5196c2 (table test-table [id=48928b185b5249b9b2acbdada1326caf]) successfully deleted
I20260926 13:45:02.726217 14681 catalog_manager.cc:4841] TS a1299c2a5fc0416f8d90363abdb436e2 (127.12.170.65:41243): tablet 8bce5014ff454322be85812a6d5196c2 (table test-table [id=48928b185b5249b9b2acbdada1326caf]) successfully deleted
I20260926 13:45:02.728206 15285 ts_tablet_manager.cc:1918] T 8bce5014ff454322be85812a6d5196c2 P e41140ad3adc462ea3e3b92afebdf016: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 13:45:02.728267 15285 log.cc:1196] T 8bce5014ff454322be85812a6d5196c2 P e41140ad3adc462ea3e3b92afebdf016: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-1/wal/wals/8bce5014ff454322be85812a6d5196c2
I20260926 13:45:02.728466 15285 ts_tablet_manager.cc:1939] T 8bce5014ff454322be85812a6d5196c2 P e41140ad3adc462ea3e3b92afebdf016: Deleting consensus metadata
I20260926 13:45:02.729046 14683 catalog_manager.cc:4841] TS e41140ad3adc462ea3e3b92afebdf016 (127.12.170.66:43493): tablet 8bce5014ff454322be85812a6d5196c2 (table test-table [id=48928b185b5249b9b2acbdada1326caf]) successfully deleted
Sep 26 13:45:02 dist-test-slave-hc1k krb5kdc[14652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430302, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 13:45:02 dist-test-slave-hc1k krb5kdc[14652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430302, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:02.749637 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:02.738630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38500 (local address 127.12.170.126:46601)
0926 13:45:02.738856 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:02.738862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:02.738906 (+ 44us) server_negotiation.cc:407] Connection header received
0926 13:45:02.739016 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:02.739018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:02.739056 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:02.739117 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:02.739704 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:02.740236 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:02.740977 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:02.741201 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:02.741264 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:02.746823 (+ 5559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:02.746826 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:02.746828 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:02.746849 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:02.748307 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:02.748684 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:02.748686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:02.748688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:02.748760 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:02.748942 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:02.748944 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:02.748945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:02.749105 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:02.749201 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:02.749375 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 13:45:02.749467 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":77,"threads_started":1}
I20260926 13:45:02.751806 14695 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:45:02.752185 14695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:45:02.756363 14834 tablet_service.cc:1467] Processing CreateTablet for tablet 032d12554cc145a7a6c7d7ac926909d3 (DEFAULT_TABLE table=test-table [id=b7512d72e9464a3695556de437eec3ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:02.756421 14974 tablet_service.cc:1467] Processing CreateTablet for tablet 032d12554cc145a7a6c7d7ac926909d3 (DEFAULT_TABLE table=test-table [id=b7512d72e9464a3695556de437eec3ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:02.756635 14834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 032d12554cc145a7a6c7d7ac926909d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:02.756594 15113 tablet_service.cc:1467] Processing CreateTablet for tablet 032d12554cc145a7a6c7d7ac926909d3 (DEFAULT_TABLE table=test-table [id=b7512d72e9464a3695556de437eec3ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:02.756678 14974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 032d12554cc145a7a6c7d7ac926909d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:02.756814 15113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 032d12554cc145a7a6c7d7ac926909d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:02.758448 15295 tablet_bootstrap.cc:492] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016: Bootstrap starting.
I20260926 13:45:02.759135 15297 tablet_bootstrap.cc:492] T 032d12554cc145a7a6c7d7ac926909d3 P add52672f9e746059350930631bb2fe9: Bootstrap starting.
I20260926 13:45:02.759274 15295 tablet_bootstrap.cc:654] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:02.759366 15296 tablet_bootstrap.cc:492] T 032d12554cc145a7a6c7d7ac926909d3 P a1299c2a5fc0416f8d90363abdb436e2: Bootstrap starting.
I20260926 13:45:02.759908 15297 tablet_bootstrap.cc:654] T 032d12554cc145a7a6c7d7ac926909d3 P add52672f9e746059350930631bb2fe9: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:02.760180 15296 tablet_bootstrap.cc:654] T 032d12554cc145a7a6c7d7ac926909d3 P a1299c2a5fc0416f8d90363abdb436e2: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:02.760237 15295 tablet_bootstrap.cc:492] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016: No bootstrap required, opened a new log
I20260926 13:45:02.760313 15295 ts_tablet_manager.cc:1397] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:02.760697 15295 raft_consensus.cc:348] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:02.760802 15295 raft_consensus.cc:374] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:02.760833 15295 raft_consensus.cc:729] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41140ad3adc462ea3e3b92afebdf016, State: Initialized, Role: FOLLOWER
I20260926 13:45:02.760948 15295 consensus_queue.cc:260] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:02.761111 15297 tablet_bootstrap.cc:492] T 032d12554cc145a7a6c7d7ac926909d3 P add52672f9e746059350930631bb2fe9: No bootstrap required, opened a new log
I20260926 13:45:02.761178 15297 ts_tablet_manager.cc:1397] T 032d12554cc145a7a6c7d7ac926909d3 P add52672f9e746059350930631bb2fe9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:02.761276 15295 ts_tablet_manager.cc:1428] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:45:02.761436 15296 tablet_bootstrap.cc:492] T 032d12554cc145a7a6c7d7ac926909d3 P a1299c2a5fc0416f8d90363abdb436e2: No bootstrap required, opened a new log
I20260926 13:45:02.761503 15296 ts_tablet_manager.cc:1397] T 032d12554cc145a7a6c7d7ac926909d3 P a1299c2a5fc0416f8d90363abdb436e2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:02.761525 15297 raft_consensus.cc:348] T 032d12554cc145a7a6c7d7ac926909d3 P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:02.761598 15297 raft_consensus.cc:374] T 032d12554cc145a7a6c7d7ac926909d3 P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:02.761617 15297 raft_consensus.cc:729] T 032d12554cc145a7a6c7d7ac926909d3 P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: add52672f9e746059350930631bb2fe9, State: Initialized, Role: FOLLOWER
I20260926 13:45:02.761719 15297 consensus_queue.cc:260] T 032d12554cc145a7a6c7d7ac926909d3 P add52672f9e746059350930631bb2fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:02.761951 15297 ts_tablet_manager.cc:1428] T 032d12554cc145a7a6c7d7ac926909d3 P add52672f9e746059350930631bb2fe9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 13:45:02.762199 15296 raft_consensus.cc:348] T 032d12554cc145a7a6c7d7ac926909d3 P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:02.762310 15296 raft_consensus.cc:374] T 032d12554cc145a7a6c7d7ac926909d3 P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:02.762348 15296 raft_consensus.cc:729] T 032d12554cc145a7a6c7d7ac926909d3 P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1299c2a5fc0416f8d90363abdb436e2, State: Initialized, Role: FOLLOWER
I20260926 13:45:02.762465 15296 consensus_queue.cc:260] T 032d12554cc145a7a6c7d7ac926909d3 P a1299c2a5fc0416f8d90363abdb436e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:02.762813 15296 ts_tablet_manager.cc:1428] T 032d12554cc145a7a6c7d7ac926909d3 P a1299c2a5fc0416f8d90363abdb436e2: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 13:45:02.840229 15301 raft_consensus.cc:482] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:45:02.840363 15301 raft_consensus.cc:504] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:02.840737 15301 leader_election.cc:288] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1299c2a5fc0416f8d90363abdb436e2 (127.12.170.65:41243), add52672f9e746059350930631bb2fe9 (127.12.170.67:38353)
I20260926 13:45:02.841125 15134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "032d12554cc145a7a6c7d7ac926909d3" candidate_uuid: "e41140ad3adc462ea3e3b92afebdf016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "add52672f9e746059350930631bb2fe9" is_pre_election: true
I20260926 13:45:02.841140 14853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "032d12554cc145a7a6c7d7ac926909d3" candidate_uuid: "e41140ad3adc462ea3e3b92afebdf016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1299c2a5fc0416f8d90363abdb436e2" is_pre_election: true
I20260926 13:45:02.841274 15134 raft_consensus.cc:2454] T 032d12554cc145a7a6c7d7ac926909d3 P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e41140ad3adc462ea3e3b92afebdf016 in term 0.
I20260926 13:45:02.841274 14853 raft_consensus.cc:2454] T 032d12554cc145a7a6c7d7ac926909d3 P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e41140ad3adc462ea3e3b92afebdf016 in term 0.
I20260926 13:45:02.841570 14926 leader_election.cc:302] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: add52672f9e746059350930631bb2fe9, e41140ad3adc462ea3e3b92afebdf016; no voters:
I20260926 13:45:02.841740 15301 raft_consensus.cc:2781] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:45:02.841809 15301 raft_consensus.cc:482] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:45:02.841836 15301 raft_consensus.cc:3037] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:02.842739 15301 raft_consensus.cc:504] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:02.843019 15301 leader_election.cc:288] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016 [CANDIDATE]: Term 1 election: Requested vote from peers a1299c2a5fc0416f8d90363abdb436e2 (127.12.170.65:41243), add52672f9e746059350930631bb2fe9 (127.12.170.67:38353)
I20260926 13:45:02.843323 14853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "032d12554cc145a7a6c7d7ac926909d3" candidate_uuid: "e41140ad3adc462ea3e3b92afebdf016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1299c2a5fc0416f8d90363abdb436e2"
I20260926 13:45:02.843364 15134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "032d12554cc145a7a6c7d7ac926909d3" candidate_uuid: "e41140ad3adc462ea3e3b92afebdf016" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "add52672f9e746059350930631bb2fe9"
I20260926 13:45:02.843434 14853 raft_consensus.cc:3037] T 032d12554cc145a7a6c7d7ac926909d3 P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:02.843461 15134 raft_consensus.cc:3037] T 032d12554cc145a7a6c7d7ac926909d3 P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:02.844339 15134 raft_consensus.cc:2454] T 032d12554cc145a7a6c7d7ac926909d3 P add52672f9e746059350930631bb2fe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e41140ad3adc462ea3e3b92afebdf016 in term 1.
I20260926 13:45:02.844388 14853 raft_consensus.cc:2454] T 032d12554cc145a7a6c7d7ac926909d3 P a1299c2a5fc0416f8d90363abdb436e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e41140ad3adc462ea3e3b92afebdf016 in term 1.
I20260926 13:45:02.844575 14926 leader_election.cc:302] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: add52672f9e746059350930631bb2fe9, e41140ad3adc462ea3e3b92afebdf016; no voters:
I20260926 13:45:02.844769 15301 raft_consensus.cc:2781] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:02.844947 15301 raft_consensus.cc:686] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016 [term 1 LEADER]: Becoming Leader. State: Replica: e41140ad3adc462ea3e3b92afebdf016, State: Running, Role: LEADER
I20260926 13:45:02.845084 15301 consensus_queue.cc:237] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:02.846565 14697 catalog_manager.cc:5495] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016 reported cstate change: term changed from 0 to 1, leader changed from <none> to e41140ad3adc462ea3e3b92afebdf016 (127.12.170.66). New cstate: current_term: 1 leader_uuid: "e41140ad3adc462ea3e3b92afebdf016" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } health_report { overall_health: UNKNOWN } } }
I20260926 13:45:02.863476 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:02.860798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44634 (local address 127.12.170.66:43493)
0926 13:45:02.861057 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:02.861064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:02.861074 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:45:02.861108 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:02.861110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:02.861148 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:02.861213 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:45:02.861396 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:02.861899 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:02.862657 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:02.862870 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:02.862939 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:02.863133 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:45:02.863196 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:45:02.863242 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:02.863302 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 13:45:02.863343 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":113,"threads_started":1}
I20260926 13:45:02.868603 14853 raft_consensus.cc:1261] T 032d12554cc145a7a6c7d7ac926909d3 P a1299c2a5fc0416f8d90363abdb436e2 [term 1 FOLLOWER]: Refusing update from remote peer e41140ad3adc462ea3e3b92afebdf016: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:02.868672 15134 raft_consensus.cc:1261] T 032d12554cc145a7a6c7d7ac926909d3 P add52672f9e746059350930631bb2fe9 [term 1 FOLLOWER]: Refusing update from remote peer e41140ad3adc462ea3e3b92afebdf016: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:02.868942 15303 consensus_queue.cc:1035] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:02.869094 15301 consensus_queue.cc:1035] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016 [LEADER]: Connected to new peer: Peer: permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:02.898880 14695 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:45:02.899035 14695 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:45:02.900647 14695 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4: Sending DeleteTablet for 3 replicas of tablet 032d12554cc145a7a6c7d7ac926909d3
I20260926 13:45:02.901106 14834 tablet_service.cc:1514] Processing DeleteTablet for tablet 032d12554cc145a7a6c7d7ac926909d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:02 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:43268
I20260926 13:45:02.901124 14974 tablet_service.cc:1514] Processing DeleteTablet for tablet 032d12554cc145a7a6c7d7ac926909d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:02 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:44608
I20260926 13:45:02.901186 15113 tablet_service.cc:1514] Processing DeleteTablet for tablet 032d12554cc145a7a6c7d7ac926909d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:02 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:39642
I20260926 13:45:02.901342 15284 tablet_replica.cc:330] T 032d12554cc145a7a6c7d7ac926909d3 P a1299c2a5fc0416f8d90363abdb436e2: stopping tablet replica
I20260926 13:45:02.901343 15286 tablet_replica.cc:330] T 032d12554cc145a7a6c7d7ac926909d3 P add52672f9e746059350930631bb2fe9: stopping tablet replica
I20260926 13:45:02.901453 15284 raft_consensus.cc:2229] T 032d12554cc145a7a6c7d7ac926909d3 P a1299c2a5fc0416f8d90363abdb436e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:45:02.901453 15286 raft_consensus.cc:2229] T 032d12554cc145a7a6c7d7ac926909d3 P add52672f9e746059350930631bb2fe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:45:02.901469 15285 tablet_replica.cc:330] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016: stopping tablet replica
I20260926 13:45:02.901553 15286 raft_consensus.cc:2258] T 032d12554cc145a7a6c7d7ac926909d3 P add52672f9e746059350930631bb2fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:02.901577 15285 raft_consensus.cc:2229] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:45:02.901582 15284 raft_consensus.cc:2258] T 032d12554cc145a7a6c7d7ac926909d3 P a1299c2a5fc0416f8d90363abdb436e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:02.901750 15285 raft_consensus.cc:2258] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:02.902190 15284 ts_tablet_manager.cc:1905] T 032d12554cc145a7a6c7d7ac926909d3 P a1299c2a5fc0416f8d90363abdb436e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:02.902190 15286 ts_tablet_manager.cc:1905] T 032d12554cc145a7a6c7d7ac926909d3 P add52672f9e746059350930631bb2fe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:02.902334 15285 ts_tablet_manager.cc:1905] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:02.902365 14695 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:45:02.902730 14695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:45:02.904609 15286 ts_tablet_manager.cc:1918] T 032d12554cc145a7a6c7d7ac926909d3 P add52672f9e746059350930631bb2fe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:45:02.904682 15286 log.cc:1196] T 032d12554cc145a7a6c7d7ac926909d3 P add52672f9e746059350930631bb2fe9: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/032d12554cc145a7a6c7d7ac926909d3
I20260926 13:45:02.904671 15284 ts_tablet_manager.cc:1918] T 032d12554cc145a7a6c7d7ac926909d3 P a1299c2a5fc0416f8d90363abdb436e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:45:02.904700 15285 ts_tablet_manager.cc:1918] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:45:02.904724 15284 log.cc:1196] T 032d12554cc145a7a6c7d7ac926909d3 P a1299c2a5fc0416f8d90363abdb436e2: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-0/wal/wals/032d12554cc145a7a6c7d7ac926909d3
I20260926 13:45:02.904767 15285 log.cc:1196] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-1/wal/wals/032d12554cc145a7a6c7d7ac926909d3
I20260926 13:45:02.904914 15286 ts_tablet_manager.cc:1939] T 032d12554cc145a7a6c7d7ac926909d3 P add52672f9e746059350930631bb2fe9: Deleting consensus metadata
I20260926 13:45:02.904929 15284 ts_tablet_manager.cc:1939] T 032d12554cc145a7a6c7d7ac926909d3 P a1299c2a5fc0416f8d90363abdb436e2: Deleting consensus metadata
I20260926 13:45:02.905068 15285 ts_tablet_manager.cc:1939] T 032d12554cc145a7a6c7d7ac926909d3 P e41140ad3adc462ea3e3b92afebdf016: Deleting consensus metadata
I20260926 13:45:02.905598 14681 catalog_manager.cc:4841] TS a1299c2a5fc0416f8d90363abdb436e2 (127.12.170.65:41243): tablet 032d12554cc145a7a6c7d7ac926909d3 (table test-table [id=b7512d72e9464a3695556de437eec3ff]) successfully deleted
I20260926 13:45:02.905598 14682 catalog_manager.cc:4841] TS add52672f9e746059350930631bb2fe9 (127.12.170.67:38353): tablet 032d12554cc145a7a6c7d7ac926909d3 (table test-table [id=b7512d72e9464a3695556de437eec3ff]) successfully deleted
I20260926 13:45:02.905843 14683 catalog_manager.cc:4841] TS e41140ad3adc462ea3e3b92afebdf016 (127.12.170.66:43493): tablet 032d12554cc145a7a6c7d7ac926909d3 (table test-table [id=b7512d72e9464a3695556de437eec3ff]) successfully deleted
I20260926 13:45:02.906919 14834 tablet_service.cc:1467] Processing CreateTablet for tablet c86caadd73134685a850b726f7e559a1 (DEFAULT_TABLE table=test-table [id=af45c1961b214ea79f2095fb240083b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:02.907001 14974 tablet_service.cc:1467] Processing CreateTablet for tablet c86caadd73134685a850b726f7e559a1 (DEFAULT_TABLE table=test-table [id=af45c1961b214ea79f2095fb240083b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:02.907176 14834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86caadd73134685a850b726f7e559a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:02.907218 14974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86caadd73134685a850b726f7e559a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:02.908965 15296 tablet_bootstrap.cc:492] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2: Bootstrap starting.
I20260926 13:45:02.909029 15295 tablet_bootstrap.cc:492] T c86caadd73134685a850b726f7e559a1 P e41140ad3adc462ea3e3b92afebdf016: Bootstrap starting.
I20260926 13:45:02.909471 15113 tablet_service.cc:1467] Processing CreateTablet for tablet c86caadd73134685a850b726f7e559a1 (DEFAULT_TABLE table=test-table [id=af45c1961b214ea79f2095fb240083b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:02.909713 15113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86caadd73134685a850b726f7e559a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:02.909751 15295 tablet_bootstrap.cc:654] T c86caadd73134685a850b726f7e559a1 P e41140ad3adc462ea3e3b92afebdf016: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:02.909829 15296 tablet_bootstrap.cc:654] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:02.910670 15295 tablet_bootstrap.cc:492] T c86caadd73134685a850b726f7e559a1 P e41140ad3adc462ea3e3b92afebdf016: No bootstrap required, opened a new log
I20260926 13:45:02.910745 15295 ts_tablet_manager.cc:1397] T c86caadd73134685a850b726f7e559a1 P e41140ad3adc462ea3e3b92afebdf016: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:02.911114 15295 raft_consensus.cc:348] T c86caadd73134685a850b726f7e559a1 P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:02.911227 15295 raft_consensus.cc:374] T c86caadd73134685a850b726f7e559a1 P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:02.911263 15295 raft_consensus.cc:729] T c86caadd73134685a850b726f7e559a1 P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41140ad3adc462ea3e3b92afebdf016, State: Initialized, Role: FOLLOWER
I20260926 13:45:02.911378 15295 consensus_queue.cc:260] T c86caadd73134685a850b726f7e559a1 P e41140ad3adc462ea3e3b92afebdf016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:02.911625 15295 ts_tablet_manager.cc:1428] T c86caadd73134685a850b726f7e559a1 P e41140ad3adc462ea3e3b92afebdf016: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 13:45:02.911804 15297 tablet_bootstrap.cc:492] T c86caadd73134685a850b726f7e559a1 P add52672f9e746059350930631bb2fe9: Bootstrap starting.
I20260926 13:45:02.912510 15297 tablet_bootstrap.cc:654] T c86caadd73134685a850b726f7e559a1 P add52672f9e746059350930631bb2fe9: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:02.912530 15296 tablet_bootstrap.cc:492] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2: No bootstrap required, opened a new log
I20260926 13:45:02.912600 15296 ts_tablet_manager.cc:1397] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 13:45:02.912968 15296 raft_consensus.cc:348] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:02.913089 15296 raft_consensus.cc:374] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:02.913129 15296 raft_consensus.cc:729] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1299c2a5fc0416f8d90363abdb436e2, State: Initialized, Role: FOLLOWER
I20260926 13:45:02.913252 15296 consensus_queue.cc:260] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:02.913473 15296 ts_tablet_manager.cc:1428] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 13:45:02.913918 15297 tablet_bootstrap.cc:492] T c86caadd73134685a850b726f7e559a1 P add52672f9e746059350930631bb2fe9: No bootstrap required, opened a new log
I20260926 13:45:02.913995 15297 ts_tablet_manager.cc:1397] T c86caadd73134685a850b726f7e559a1 P add52672f9e746059350930631bb2fe9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:02.914335 15297 raft_consensus.cc:348] T c86caadd73134685a850b726f7e559a1 P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:02.914420 15297 raft_consensus.cc:374] T c86caadd73134685a850b726f7e559a1 P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:02.914440 15297 raft_consensus.cc:729] T c86caadd73134685a850b726f7e559a1 P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: add52672f9e746059350930631bb2fe9, State: Initialized, Role: FOLLOWER
I20260926 13:45:02.914531 15297 consensus_queue.cc:260] T c86caadd73134685a850b726f7e559a1 P add52672f9e746059350930631bb2fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:02.914767 15297 ts_tablet_manager.cc:1428] T c86caadd73134685a850b726f7e559a1 P add52672f9e746059350930631bb2fe9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 13:45:02.922520 15302 raft_consensus.cc:482] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:45:02.922601 15302 raft_consensus.cc:504] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:02.923528 15302 leader_election.cc:288] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e41140ad3adc462ea3e3b92afebdf016 (127.12.170.66:43493), add52672f9e746059350930631bb2fe9 (127.12.170.67:38353)
I20260926 13:45:02.928340 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:02.923756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:41807 (local address 127.12.170.66:43493)
0926 13:45:02.923869 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:02.923874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:02.924065 (+ 191us) server_negotiation.cc:407] Connection header received
0926 13:45:02.924207 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:02.924210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:02.924249 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:02.924315 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:02.924663 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:02.925429 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:02.927123 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:02.927788 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:02.927902 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:02.927951 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:02.928156 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 13:45:02.928225 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 13:45:02.928344 15316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:02.923787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:60551 (local address 127.12.170.67:38353)
0926 13:45:02.923977 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:02.923987 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:02.924056 (+ 69us) server_negotiation.cc:407] Connection header received
0926 13:45:02.924193 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:02.924195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:02.924237 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:02.924315 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:02.924663 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:02.925455 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:02.927123 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:02.927807 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:02.927902 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:02.927951 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:02.928157 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 13:45:02.928228 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":60,"threads_started":1}
I20260926 13:45:02.928411 15315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:02.923735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:38353 (local address 127.12.170.65:60551)
0926 13:45:02.923973 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:02.923984 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:02.924104 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:02.924329 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:02.924332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:02.924354 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:02.924571 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:02.924576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:02.925543 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:02.925547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:02.927010 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:02.927020 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:02.927893 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:02.927898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:02.928030 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:02.928056 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 13:45:02.928128 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 13:45:02.928230 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":166,"spinlock_wait_cycles":24832,"thread_start_us":99,"threads_started":1}
I20260926 13:45:02.928438 15314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:02.923688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:43493 (local address 127.12.170.65:41807)
0926 13:45:02.923949 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:02.923962 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:02.924104 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:02.924329 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:02.924332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:02.924354 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:02.924566 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:02.924574 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:02.925558 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:02.925561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:02.927025 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:02.927032 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:02.927894 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:02.927898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:02.928055 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:02.928072 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 13:45:02.928134 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 13:45:02.928230 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":186,"spinlock_wait_cycles":7680,"thread_start_us":58,"threads_started":1}
I20260926 13:45:02.928742 14994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86caadd73134685a850b726f7e559a1" candidate_uuid: "a1299c2a5fc0416f8d90363abdb436e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41140ad3adc462ea3e3b92afebdf016" is_pre_election: true
I20260926 13:45:02.928855 14994 raft_consensus.cc:2454] T c86caadd73134685a850b726f7e559a1 P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1299c2a5fc0416f8d90363abdb436e2 in term 0.
I20260926 13:45:02.929068 15134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86caadd73134685a850b726f7e559a1" candidate_uuid: "a1299c2a5fc0416f8d90363abdb436e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "add52672f9e746059350930631bb2fe9" is_pre_election: true
I20260926 13:45:02.929174 15134 raft_consensus.cc:2454] T c86caadd73134685a850b726f7e559a1 P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1299c2a5fc0416f8d90363abdb436e2 in term 0.
I20260926 13:45:02.929190 14787 leader_election.cc:302] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1299c2a5fc0416f8d90363abdb436e2, e41140ad3adc462ea3e3b92afebdf016; no voters:
I20260926 13:45:02.929436 15302 raft_consensus.cc:2781] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:45:02.929490 15302 raft_consensus.cc:482] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:45:02.929517 15302 raft_consensus.cc:3037] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:02.930748 15302 raft_consensus.cc:504] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:02.931058 15302 leader_election.cc:288] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2 [CANDIDATE]: Term 1 election: Requested vote from peers e41140ad3adc462ea3e3b92afebdf016 (127.12.170.66:43493), add52672f9e746059350930631bb2fe9 (127.12.170.67:38353)
I20260926 13:45:02.931314 14994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86caadd73134685a850b726f7e559a1" candidate_uuid: "a1299c2a5fc0416f8d90363abdb436e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e41140ad3adc462ea3e3b92afebdf016"
I20260926 13:45:02.931336 15134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86caadd73134685a850b726f7e559a1" candidate_uuid: "a1299c2a5fc0416f8d90363abdb436e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "add52672f9e746059350930631bb2fe9"
I20260926 13:45:02.931416 15134 raft_consensus.cc:3037] T c86caadd73134685a850b726f7e559a1 P add52672f9e746059350930631bb2fe9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:02.931411 14994 raft_consensus.cc:3037] T c86caadd73134685a850b726f7e559a1 P e41140ad3adc462ea3e3b92afebdf016 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:02.932405 15134 raft_consensus.cc:2454] T c86caadd73134685a850b726f7e559a1 P add52672f9e746059350930631bb2fe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1299c2a5fc0416f8d90363abdb436e2 in term 1.
I20260926 13:45:02.932466 14994 raft_consensus.cc:2454] T c86caadd73134685a850b726f7e559a1 P e41140ad3adc462ea3e3b92afebdf016 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1299c2a5fc0416f8d90363abdb436e2 in term 1.
I20260926 13:45:02.932716 14787 leader_election.cc:302] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a1299c2a5fc0416f8d90363abdb436e2, add52672f9e746059350930631bb2fe9, e41140ad3adc462ea3e3b92afebdf016; no voters:
I20260926 13:45:02.932870 15302 raft_consensus.cc:2781] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:02.933101 15302 raft_consensus.cc:686] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2 [term 1 LEADER]: Becoming Leader. State: Replica: a1299c2a5fc0416f8d90363abdb436e2, State: Running, Role: LEADER
I20260926 13:45:02.933321 15302 consensus_queue.cc:237] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } }
I20260926 13:45:02.935160 14695 catalog_manager.cc:5495] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1299c2a5fc0416f8d90363abdb436e2 (127.12.170.65). New cstate: current_term: 1 leader_uuid: "a1299c2a5fc0416f8d90363abdb436e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1299c2a5fc0416f8d90363abdb436e2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 } health_report { overall_health: UNKNOWN } } }
I20260926 13:45:02.948778 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:02.945480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.12.170.65:41243)
0926 13:45:02.945704 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:02.945710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:02.945721 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:45:02.945775 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:02.945776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:02.945820 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:02.945926 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:45:02.946118 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:02.946694 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:02.947563 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:02.947829 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:02.947922 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:02.948106 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:45:02.948507 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:45:02.948594 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:02.948633 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 13:45:02.948672 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":119,"threads_started":1}
I20260926 13:45:02.957131 15316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:02.954548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:52753 (local address 127.12.170.67:38353)
0926 13:45:02.954643 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:02.954646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:02.954752 (+ 106us) server_negotiation.cc:407] Connection header received
0926 13:45:02.954855 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:02.954857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:02.954892 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:02.954941 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:45:02.955108 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:02.955581 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:02.956364 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:02.956574 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:02.956642 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:02.956848 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:45:02.956904 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:45:02.956948 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:02.957006 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 13:45:02.957044 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 13:45:02.957180 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:02.954424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:38353 (local address 127.12.170.65:52753)
0926 13:45:02.954693 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:02.954705 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:02.954778 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:02.954951 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:02.954953 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:02.954958 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:45:02.955042 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:02.955046 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:02.955685 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:02.955688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:02.956277 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:02.956284 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:02.956664 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:02.956666 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:02.956729 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:02.956761 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:45:02.956954 (+ 193us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:45:02.956958 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:45:02.956996 (+ 38us) client_negotiation.cc:240] Negotiation successful
0926 13:45:02.957068 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":95,"threads_started":1}
I20260926 13:45:02.961814 14994 raft_consensus.cc:1261] T c86caadd73134685a850b726f7e559a1 P e41140ad3adc462ea3e3b92afebdf016 [term 1 FOLLOWER]: Refusing update from remote peer a1299c2a5fc0416f8d90363abdb436e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:02.961871 15134 raft_consensus.cc:1261] T c86caadd73134685a850b726f7e559a1 P add52672f9e746059350930631bb2fe9 [term 1 FOLLOWER]: Refusing update from remote peer a1299c2a5fc0416f8d90363abdb436e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:02.962180 15317 consensus_queue.cc:1035] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e41140ad3adc462ea3e3b92afebdf016" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:02.962324 15302 consensus_queue.cc:1035] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "add52672f9e746059350930631bb2fe9" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 38353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:02.976790 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:02.973909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:49939 (local address 127.12.170.65:41243)
0926 13:45:02.974009 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:02.974013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:02.974173 (+ 160us) server_negotiation.cc:407] Connection header received
0926 13:45:02.974275 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:02.974278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:02.974320 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:02.974380 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:45:02.974586 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:02.975065 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:02.975889 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:02.976121 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:02.976222 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:02.976417 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:45:02.976495 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:45:02.976554 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:02.976640 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 13:45:02.976690 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 13:45:02.976847 15327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:02.973856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:41243 (local address 127.12.170.67:49939)
0926 13:45:02.974103 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:02.974114 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:02.974199 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:02.974398 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:02.974401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:02.974408 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:45:02.974501 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:02.974506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:02.975161 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:02.975164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:02.975793 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:02.975801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:02.976210 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:02.976213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:02.976283 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:02.976315 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:45:02.976564 (+ 249us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:45:02.976570 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 13:45:02.976630 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 13:45:02.976707 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":108,"threads_started":1}
W20260926 13:45:02.977391 15269 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 13:45:04.527554 14695 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:45:04.527724 14695 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:45:04.530069 14695 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef95c2a78e3449aea15ee4a910da5aa4: Sending DeleteTablet for 3 replicas of tablet c86caadd73134685a850b726f7e559a1
I20260926 13:45:04.530575 14834 tablet_service.cc:1514] Processing DeleteTablet for tablet c86caadd73134685a850b726f7e559a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:04 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:43268
I20260926 13:45:04.530629 15113 tablet_service.cc:1514] Processing DeleteTablet for tablet c86caadd73134685a850b726f7e559a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:04 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:39642
I20260926 13:45:04.530877 15351 tablet_replica.cc:330] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2: stopping tablet replica
I20260926 13:45:04.530912 14974 tablet_service.cc:1514] Processing DeleteTablet for tablet c86caadd73134685a850b726f7e559a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:04 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:44608
I20260926 13:45:04.530995 15351 raft_consensus.cc:2229] T c86caadd73134685a850b726f7e559a1 P a1299c2a5fc0416f8d90363abdb436e2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:45:04.531107 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 14769
I20260926 13:45:04.531716 15353 tablet_replica.cc:330] T c86caadd73134685a850b726f7e559a1 P e41140ad3adc462ea3e3b92afebdf016: stopping tablet replica
I20260926 13:45:04.531942 15352 tablet_replica.cc:330] T c86caadd73134685a850b726f7e559a1 P add52672f9e746059350930631bb2fe9: stopping tablet replica
I20260926 13:45:04.531941 15353 raft_consensus.cc:2229] T c86caadd73134685a850b726f7e559a1 P e41140ad3adc462ea3e3b92afebdf016 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:45:04.532044 15353 raft_consensus.cc:2258] T c86caadd73134685a850b726f7e559a1 P e41140ad3adc462ea3e3b92afebdf016 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:04.532070 15352 raft_consensus.cc:2229] T c86caadd73134685a850b726f7e559a1 P add52672f9e746059350930631bb2fe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:45:04.532172 15352 raft_consensus.cc:2258] T c86caadd73134685a850b726f7e559a1 P add52672f9e746059350930631bb2fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:04.532562 15353 ts_tablet_manager.cc:1905] T c86caadd73134685a850b726f7e559a1 P e41140ad3adc462ea3e3b92afebdf016: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:04.532766 15352 ts_tablet_manager.cc:1905] T c86caadd73134685a850b726f7e559a1 P add52672f9e746059350930631bb2fe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:04.534541 15353 ts_tablet_manager.cc:1918] T c86caadd73134685a850b726f7e559a1 P e41140ad3adc462ea3e3b92afebdf016: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 13:45:04.534590 15353 log.cc:1196] T c86caadd73134685a850b726f7e559a1 P e41140ad3adc462ea3e3b92afebdf016: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-1/wal/wals/c86caadd73134685a850b726f7e559a1
I20260926 13:45:04.534749 15353 ts_tablet_manager.cc:1939] T c86caadd73134685a850b726f7e559a1 P e41140ad3adc462ea3e3b92afebdf016: Deleting consensus metadata
I20260926 13:45:04.535367 14683 catalog_manager.cc:4841] TS e41140ad3adc462ea3e3b92afebdf016 (127.12.170.66:43493): tablet c86caadd73134685a850b726f7e559a1 (table test-table [id=af45c1961b214ea79f2095fb240083b3]) successfully deleted
I20260926 13:45:04.535451 15352 ts_tablet_manager.cc:1918] T c86caadd73134685a850b726f7e559a1 P add52672f9e746059350930631bb2fe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 13:45:04.535519 15352 log.cc:1196] T c86caadd73134685a850b726f7e559a1 P add52672f9e746059350930631bb2fe9: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/c86caadd73134685a850b726f7e559a1
I20260926 13:45:04.535821 15352 ts_tablet_manager.cc:1939] T c86caadd73134685a850b726f7e559a1 P add52672f9e746059350930631bb2fe9: Deleting consensus metadata
I20260926 13:45:04.536461 14682 catalog_manager.cc:4841] TS add52672f9e746059350930631bb2fe9 (127.12.170.67:38353): tablet c86caadd73134685a850b726f7e559a1 (table test-table [id=af45c1961b214ea79f2095fb240083b3]) successfully deleted
W20260926 13:45:04.539176 14681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.170.65:41243 (error 108)
I20260926 13:45:04.539371 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 14909
I20260926 13:45:04.541167 15355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:04.540733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:41243 (local address 127.0.0.1:43292)
0926 13:45:04.540994 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:04.541072 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.65:41243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":87,"threads_started":1}
W20260926 13:45:04.541402 14681 catalog_manager.cc:4584] TS a1299c2a5fc0416f8d90363abdb436e2 (127.12.170.65:41243): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c86caadd73134685a850b726f7e559a1: Network error: Client connection negotiation failed: client connection to 127.12.170.65:41243: connect: Connection refused (error 111)
I20260926 13:45:04.544867 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 15049
I20260926 13:45:04.550084 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 14665
2026-09-26T13:45:04Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6141 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:45:04 dist-test-slave-hc1k krb5kdc[15358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:45:04 dist-test-slave-hc1k krb5kdc[15358](info): set up 2 sockets
Sep 26 13:45:04 dist-test-slave-hc1k krb5kdc[15358](info): commencing operation
krb5kdc: starting...
W20260926 13:45:05.515213 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:45:05 dist-test-slave-hc1k krb5kdc[15358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:45:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:45:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:45:05.620378 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:38925
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:36067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:38925
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:05.720976 15371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:05.721205 15371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:05.721257 15371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:05.725109 15371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:45:05.725157 15371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:05.725183 15371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:05.725204 15371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:45:05.725232 15371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:45:05.730077 15371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:36067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:38925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:38925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:05.730919 15371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:05.731845 15371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:05.739400 15379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:05.739394 15376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:05.739394 15377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:05.739674 15371 server_base.cc:1034] running on GCE node
I20260926 13:45:05.740149 15371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:05.741065 15371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:05.742219 15371 hybrid_clock.cc:648] HybridClock initialized: now 1790430305742204 us; error 25 us; skew 500 ppm
Sep 26 13:45:05 dist-test-slave-hc1k krb5kdc[15358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430305, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:05.744719 15371 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:45:05.745424 15371 webserver.cc:460] Webserver started at http://127.12.170.126:40647/ using document root <none> and password file <none>
I20260926 13:45:05.745872 15371 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:05.745922 15371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:05.746070 15371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:05.747880 15371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "67354b62bfb04dc98a2a43ee4768079e"
format_stamp: "Formatted at 2026-09-26 13:45:05 on dist-test-slave-hc1k"
server_key: "c6b29315da87871e8304ec50ac67fe9b"
server_key_iv: "06e740823714a7145e57beea51beab8e"
server_key_version: "encryptionkey@0"
I20260926 13:45:05.748335 15371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "67354b62bfb04dc98a2a43ee4768079e"
format_stamp: "Formatted at 2026-09-26 13:45:05 on dist-test-slave-hc1k"
server_key: "c6b29315da87871e8304ec50ac67fe9b"
server_key_iv: "06e740823714a7145e57beea51beab8e"
server_key_version: "encryptionkey@0"
I20260926 13:45:05.751441 15371 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:45:05.753509 15386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:05.754590 15371 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:05.754681 15371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "67354b62bfb04dc98a2a43ee4768079e"
format_stamp: "Formatted at 2026-09-26 13:45:05 on dist-test-slave-hc1k"
server_key: "c6b29315da87871e8304ec50ac67fe9b"
server_key_iv: "06e740823714a7145e57beea51beab8e"
server_key_version: "encryptionkey@0"
I20260926 13:45:05.754768 15371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:05.773794 15371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:05.776616 15371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:05.776778 15371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:05.782998 15371 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:38925
I20260926 13:45:05.783010 15438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:38925 every 8 connection(s)
I20260926 13:45:05.783872 15371 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:45:05.784741 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 15371
I20260926 13:45:05.784837 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:45:05.785104 12969 external_mini_cluster.cc:1428] Setting key ec98b93ff0adad34a92ec67a864dd4b1
I20260926 13:45:05.786576 15439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:05.792354 15439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67354b62bfb04dc98a2a43ee4768079e: Bootstrap starting.
Sep 26 13:45:05 dist-test-slave-hc1k krb5kdc[15358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:05.794561 15439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67354b62bfb04dc98a2a43ee4768079e: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:05.795325 15439 log.cc:824] T 00000000000000000000000000000000 P 67354b62bfb04dc98a2a43ee4768079e: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:05.796950 15439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67354b62bfb04dc98a2a43ee4768079e: No bootstrap required, opened a new log
I20260926 13:45:05.798424 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:05.786351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60010 (local address 127.12.170.126:38925)
0926 13:45:05.786723 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:05.786738 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:05.786770 (+ 32us) server_negotiation.cc:407] Connection header received
0926 13:45:05.787354 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:05.787368 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:05.787702 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:05.787963 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:05.788761 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:05.789787 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:05.790879 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:05.791363 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:05.791478 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:05.793923 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:05.793927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:05.793939 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:05.793964 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:05.795762 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:05.796178 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:05.796184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:05.796189 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:05.796294 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:05.796546 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:05.796549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:05.796551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:05.796897 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:05.797074 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:05.797338 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 13:45:05.797598 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":116,"threads_started":1}
I20260926 13:45:05.799748 15439 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67354b62bfb04dc98a2a43ee4768079e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67354b62bfb04dc98a2a43ee4768079e" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 38925 } }
I20260926 13:45:05.799942 15439 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67354b62bfb04dc98a2a43ee4768079e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:05.799997 15439 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67354b62bfb04dc98a2a43ee4768079e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67354b62bfb04dc98a2a43ee4768079e, State: Initialized, Role: FOLLOWER
I20260926 13:45:05.800384 15439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67354b62bfb04dc98a2a43ee4768079e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67354b62bfb04dc98a2a43ee4768079e" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 38925 } }
I20260926 13:45:05.800504 15439 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67354b62bfb04dc98a2a43ee4768079e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:45:05.800565 15439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67354b62bfb04dc98a2a43ee4768079e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:45:05.800655 15439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67354b62bfb04dc98a2a43ee4768079e [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:05.801390 15439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67354b62bfb04dc98a2a43ee4768079e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67354b62bfb04dc98a2a43ee4768079e" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 38925 } }
I20260926 13:45:05.801687 15439 leader_election.cc:302] T 00000000000000000000000000000000 P 67354b62bfb04dc98a2a43ee4768079e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67354b62bfb04dc98a2a43ee4768079e; no voters:
I20260926 13:45:05.801928 15439 leader_election.cc:288] T 00000000000000000000000000000000 P 67354b62bfb04dc98a2a43ee4768079e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:45:05.802018 15444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67354b62bfb04dc98a2a43ee4768079e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:05.802210 15444 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67354b62bfb04dc98a2a43ee4768079e [term 1 LEADER]: Becoming Leader. State: Replica: 67354b62bfb04dc98a2a43ee4768079e, State: Running, Role: LEADER
I20260926 13:45:05.802526 15444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67354b62bfb04dc98a2a43ee4768079e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67354b62bfb04dc98a2a43ee4768079e" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 38925 } }
I20260926 13:45:05.802958 15439 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67354b62bfb04dc98a2a43ee4768079e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:45:05.803968 15445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67354b62bfb04dc98a2a43ee4768079e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67354b62bfb04dc98a2a43ee4768079e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67354b62bfb04dc98a2a43ee4768079e" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 38925 } } }
I20260926 13:45:05.804313 15445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67354b62bfb04dc98a2a43ee4768079e [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:05.804214 15446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67354b62bfb04dc98a2a43ee4768079e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67354b62bfb04dc98a2a43ee4768079e. Latest consensus state: current_term: 1 leader_uuid: "67354b62bfb04dc98a2a43ee4768079e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67354b62bfb04dc98a2a43ee4768079e" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 38925 } } }
I20260926 13:45:05.804632 15446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67354b62bfb04dc98a2a43ee4768079e [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:05.805127 15456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:45:05.807793 15456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:45:05.812266 15456 catalog_manager.cc:1347] Generated new cluster ID: 00b1a9271627464f8b41d4b3ed0e2da3
I20260926 13:45:05.812325 15456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:45:05.825706 15456 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:45:05.826532 15456 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:45:05.838317 15456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67354b62bfb04dc98a2a43ee4768079e: Generated new TSK 0
I20260926 13:45:05.838898 15456 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.170.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
I20260926 13:45:05.864156 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:0
--local_ip_for_outbound_sockets=127.12.170.65
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:38925
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:36067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:05.966630 15465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:05.966840 15465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:05.966876 15465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:05.970749 15465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:05.970798 15465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:05.970849 15465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:45:05.975696 15465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:36067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:38925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:05.976507 15465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:05.977329 15465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:05.985769 15470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:05.985864 15473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:05.985829 15465 server_base.cc:1034] running on GCE node
W20260926 13:45:05.985919 15471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:05.986349 15465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:05.986891 15465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:05.988036 15465 hybrid_clock.cc:648] HybridClock initialized: now 1790430305988015 us; error 31 us; skew 500 ppm
Sep 26 13:45:05 dist-test-slave-hc1k krb5kdc[15358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430305, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:05.990523 15465 init.cc:377] Logged in from keytab as kudu/127.12.170.65@KRBTEST.COM (short username kudu)
I20260926 13:45:05.991257 15465 webserver.cc:460] Webserver started at http://127.12.170.65:33965/ using document root <none> and password file <none>
I20260926 13:45:05.991767 15465 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:05.991832 15465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:05.992012 15465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:05.993844 15465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/data/instance:
uuid: "a408c4b2c65f4bc0afc8ba0e5571820a"
format_stamp: "Formatted at 2026-09-26 13:45:05 on dist-test-slave-hc1k"
server_key: "dead84da6b008ac14822403faeaf380c"
server_key_iv: "0b64498f03f6caad57949f28d21317fe"
server_key_version: "encryptionkey@0"
I20260926 13:45:05.994349 15465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance:
uuid: "a408c4b2c65f4bc0afc8ba0e5571820a"
format_stamp: "Formatted at 2026-09-26 13:45:05 on dist-test-slave-hc1k"
server_key: "dead84da6b008ac14822403faeaf380c"
server_key_iv: "0b64498f03f6caad57949f28d21317fe"
server_key_version: "encryptionkey@0"
I20260926 13:45:05.997632 15465 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:45:05.999753 15480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:06.000823 15465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:06.000921 15465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "a408c4b2c65f4bc0afc8ba0e5571820a"
format_stamp: "Formatted at 2026-09-26 13:45:05 on dist-test-slave-hc1k"
server_key: "dead84da6b008ac14822403faeaf380c"
server_key_iv: "0b64498f03f6caad57949f28d21317fe"
server_key_version: "encryptionkey@0"
I20260926 13:45:06.001001 15465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:06.023301 15465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:06.026131 15465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:06.026289 15465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:06.026796 15465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:06.027549 15465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:06.027593 15465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:06.027643 15465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:06.027694 15465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:06.035730 15465 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:40335
I20260926 13:45:06.035768 15593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:40335 every 8 connection(s)
I20260926 13:45:06.036640 15465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:45:06.038976 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 15465
I20260926 13:45:06.039089 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance
I20260926 13:45:06.039369 12969 external_mini_cluster.cc:1428] Setting key f487aef0412aa0eb62086a1584851226
Sep 26 13:45:06 dist-test-slave-hc1k krb5kdc[15358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430305, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.170.66@KRBTEST.COM; defaulting to no policy
I20260926 13:45:06.048950 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:06.038253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:33559 (local address 127.12.170.126:38925)
0926 13:45:06.038376 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:06.038380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:06.039321 (+ 941us) server_negotiation.cc:407] Connection header received
0926 13:45:06.040291 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:06.040295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:06.040344 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:06.040413 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:06.041463 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:06.041980 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:06.043068 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:06.043286 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:06.043369 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:06.045820 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:06.045823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:06.045825 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:06.045845 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:06.047102 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:06.047638 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:06.047641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:06.047642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:06.047752 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:06.048024 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:06.048026 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:06.048027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:06.048192 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:06.048299 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:06.048709 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 13:45:06.048808 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.66@KRBTEST.COM" created.
I20260926 13:45:06.049801 15596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:06.038578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:38925 (local address 127.12.170.65:33559)
0926 13:45:06.039191 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:06.039234 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:06.040115 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:06.040598 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:06.040605 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:06.040912 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:06.041335 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:06.041347 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:06.042090 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:06.042093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:06.042952 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:06.042961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:06.043410 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:06.043414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:06.043570 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:06.044065 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:06.044080 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:06.045663 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:06.047208 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:06.047213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:06.047226 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:06.047552 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:06.047823 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:06.047826 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:06.047827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:06.047950 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:06.048302 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:06.048306 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:06.048525 (+ 219us) client_negotiation.cc:769] Sending connection context
0926 13:45:06.048670 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 13:45:06.048889 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":119,"threads_started":1}
I20260926 13:45:06.050855 15594 heartbeater.cc:347] Connected to a master server at 127.12.170.126:38925
I20260926 13:45:06.051085 15594 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:06.051507 15594 heartbeater.cc:511] Master 127.12.170.126:38925 requested a full tablet report, sending...
I20260926 13:45:06.052879 15403 ts_manager.cc:194] Registered new tserver with Master: a408c4b2c65f4bc0afc8ba0e5571820a (127.12.170.65:40335)
I20260926 13:45:06.053974 15403 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.65@KRBTEST.COM'} at 127.12.170.65:33559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
I20260926 13:45:06.062633 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:0
--local_ip_for_outbound_sockets=127.12.170.66
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:38925
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:36067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:06.164809 15599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:06.165042 15599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:06.165081 15599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:06.168939 15599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:06.169008 15599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:06.169083 15599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:45:06.174290 15599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:36067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:38925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:06.175256 15599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:06.176236 15599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:06.184916 15604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:06.184954 15607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:06.184954 15605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:06.185531 15599 server_base.cc:1034] running on GCE node
I20260926 13:45:06.185865 15599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:06.186453 15599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:06.187637 15599 hybrid_clock.cc:648] HybridClock initialized: now 1790430306187615 us; error 31 us; skew 500 ppm
Sep 26 13:45:06 dist-test-slave-hc1k krb5kdc[15358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430306, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:06.190243 15599 init.cc:377] Logged in from keytab as kudu/127.12.170.66@KRBTEST.COM (short username kudu)
I20260926 13:45:06.191013 15599 webserver.cc:460] Webserver started at http://127.12.170.66:37907/ using document root <none> and password file <none>
I20260926 13:45:06.191526 15599 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:06.191588 15599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:06.191804 15599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:06.194084 15599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/data/instance:
uuid: "2922c472e7df47dba2461afd0129522b"
format_stamp: "Formatted at 2026-09-26 13:45:06 on dist-test-slave-hc1k"
server_key: "fb467dacb0b0b5ee9b6a24e8503612c1"
server_key_iv: "3ce7b317e6d47c833334269992fdc0d2"
server_key_version: "encryptionkey@0"
I20260926 13:45:06.194700 15599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance:
uuid: "2922c472e7df47dba2461afd0129522b"
format_stamp: "Formatted at 2026-09-26 13:45:06 on dist-test-slave-hc1k"
server_key: "fb467dacb0b0b5ee9b6a24e8503612c1"
server_key_iv: "3ce7b317e6d47c833334269992fdc0d2"
server_key_version: "encryptionkey@0"
I20260926 13:45:06.198972 15599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 13:45:06.201166 15614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:06.202258 15599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:06.202358 15599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "2922c472e7df47dba2461afd0129522b"
format_stamp: "Formatted at 2026-09-26 13:45:06 on dist-test-slave-hc1k"
server_key: "fb467dacb0b0b5ee9b6a24e8503612c1"
server_key_iv: "3ce7b317e6d47c833334269992fdc0d2"
server_key_version: "encryptionkey@0"
I20260926 13:45:06.202484 15599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:06.233016 15599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:06.235751 15599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:06.235903 15599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:06.236414 15599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:06.237166 15599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:06.237215 15599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:06.237265 15599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:06.237289 15599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:06.245877 15599 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:33701
I20260926 13:45:06.245903 15727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:33701 every 8 connection(s)
I20260926 13:45:06.246654 15599 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:45:06.247879 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 15599
I20260926 13:45:06.247978 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance
I20260926 13:45:06.248239 12969 external_mini_cluster.cc:1428] Setting key d16c57869a9a9fc4b1400ec27a1c38eb
Sep 26 13:45:06 dist-test-slave-hc1k krb5kdc[15358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430306, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.170.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.67@KRBTEST.COM" created.
I20260926 13:45:06.260692 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:06.248406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:53953 (local address 127.12.170.126:38925)
0926 13:45:06.248566 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:06.248572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:06.249715 (+ 1143us) server_negotiation.cc:407] Connection header received
0926 13:45:06.250562 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:06.250565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:06.250615 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:06.250696 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:06.252204 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:06.252946 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:06.254089 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:06.254437 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:06.254550 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:06.257140 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:06.257144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:06.257148 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:06.257178 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:06.258635 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:06.259110 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:06.259113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:06.259114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:06.259201 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:06.259536 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:06.259538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:06.259539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:06.259793 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:06.259943 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:06.260378 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 13:45:06.260516 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 13:45:06.261421 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:06.248811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:38925 (local address 127.12.170.66:53953)
0926 13:45:06.249563 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:06.249615 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:06.250369 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:06.250881 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:06.250892 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:06.251355 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:06.252031 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:06.252052 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:06.253071 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:06.253074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:06.253962 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:06.253972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:06.254568 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:06.254570 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:06.254708 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:06.255306 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:06.255324 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:06.256951 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:06.258730 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:06.258735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:06.258745 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:06.258995 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:06.259312 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:06.259315 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:06.259316 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:06.259449 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:06.259902 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:06.259907 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:06.260138 (+ 231us) client_negotiation.cc:769] Sending connection context
0926 13:45:06.260287 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 13:45:06.260522 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":129,"threads_started":1}
I20260926 13:45:06.262557 15728 heartbeater.cc:347] Connected to a master server at 127.12.170.126:38925
I20260926 13:45:06.262776 15728 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:06.263242 15728 heartbeater.cc:511] Master 127.12.170.126:38925 requested a full tablet report, sending...
I20260926 13:45:06.264310 15403 ts_manager.cc:194] Registered new tserver with Master: 2922c472e7df47dba2461afd0129522b (127.12.170.66:33701)
I20260926 13:45:06.264859 15403 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.66@KRBTEST.COM'} at 127.12.170.66:53953
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
I20260926 13:45:06.274204 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:0
--local_ip_for_outbound_sockets=127.12.170.67
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:38925
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:36067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:06.376591 15733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:06.376813 15733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:06.376864 15733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:06.380708 15733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:06.380759 15733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:06.380831 15733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:45:06.385860 15733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:36067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:38925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:06.386715 15733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:06.387580 15733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:06.396059 15738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:06.396099 15739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:06.396315 15741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:06.396435 15733 server_base.cc:1034] running on GCE node
I20260926 13:45:06.396780 15733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:06.397337 15733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:06.398500 15733 hybrid_clock.cc:648] HybridClock initialized: now 1790430306398476 us; error 40 us; skew 500 ppm
Sep 26 13:45:06 dist-test-slave-hc1k krb5kdc[15358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430306, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:06.401098 15733 init.cc:377] Logged in from keytab as kudu/127.12.170.67@KRBTEST.COM (short username kudu)
I20260926 13:45:06.401876 15733 webserver.cc:460] Webserver started at http://127.12.170.67:35633/ using document root <none> and password file <none>
I20260926 13:45:06.402379 15733 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:06.402447 15733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:06.402657 15733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:06.404767 15733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/data/instance:
uuid: "f218139818d04711bff645d77e61f623"
format_stamp: "Formatted at 2026-09-26 13:45:06 on dist-test-slave-hc1k"
server_key: "d4bbcbe6c5e7246321fe785e104d37d8"
server_key_iv: "d6a908b0e45e17f308614a8fdc7ba93b"
server_key_version: "encryptionkey@0"
I20260926 13:45:06.405345 15733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance:
uuid: "f218139818d04711bff645d77e61f623"
format_stamp: "Formatted at 2026-09-26 13:45:06 on dist-test-slave-hc1k"
server_key: "d4bbcbe6c5e7246321fe785e104d37d8"
server_key_iv: "d6a908b0e45e17f308614a8fdc7ba93b"
server_key_version: "encryptionkey@0"
I20260926 13:45:06.408870 15733 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:45:06.411123 15748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:06.412288 15733 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:45:06.412401 15733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "f218139818d04711bff645d77e61f623"
format_stamp: "Formatted at 2026-09-26 13:45:06 on dist-test-slave-hc1k"
server_key: "d4bbcbe6c5e7246321fe785e104d37d8"
server_key_iv: "d6a908b0e45e17f308614a8fdc7ba93b"
server_key_version: "encryptionkey@0"
I20260926 13:45:06.412490 15733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:06.422796 15733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:06.425384 15733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:06.425529 15733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:06.425998 15733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:06.426687 15733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:06.426729 15733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:06.426776 15733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:06.426805 15733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:06.434360 15733 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:33733
I20260926 13:45:06.434405 15861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:33733 every 8 connection(s)
I20260926 13:45:06.435115 15733 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:45:06.438902 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 15733
I20260926 13:45:06.438993 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance
I20260926 13:45:06.439227 12969 external_mini_cluster.cc:1428] Setting key fe91e1ccefcd0e490bd452743a671df2
Sep 26 13:45:06 dist-test-slave-hc1k krb5kdc[15358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430306, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:06.446622 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:06.436636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:45637 (local address 127.12.170.126:38925)
0926 13:45:06.436747 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:06.436751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:06.437432 (+ 681us) server_negotiation.cc:407] Connection header received
0926 13:45:06.438632 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:06.438635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:06.438692 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:06.438772 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:06.439863 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:06.440609 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:06.441772 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:06.441986 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:06.442064 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:06.444096 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:06.444099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:06.444101 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:06.444123 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:06.445153 (+ 1030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:06.445546 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:06.445548 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:06.445550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:06.445610 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:06.445835 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:06.445837 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:06.445838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:06.445994 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:06.446065 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:06.446416 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 13:45:06.446511 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 13:45:06.447302 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:06.436918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:38925 (local address 127.12.170.67:45637)
0926 13:45:06.437302 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:06.437339 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:06.438442 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:06.438921 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:06.438927 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:06.439278 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:06.439746 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:06.439761 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:06.440726 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:06.440729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:06.441670 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:06.441680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:06.442093 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:06.442095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:06.442224 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:06.442657 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:06.442672 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:06.443939 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:06.445245 (+ 1306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:06.445249 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:06.445259 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:06.445476 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:06.445668 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:06.445670 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:06.445671 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:06.445778 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:06.446059 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:06.446063 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:06.446250 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 13:45:06.446385 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 13:45:06.446596 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":102,"threads_started":1}
I20260926 13:45:06.448362 15862 heartbeater.cc:347] Connected to a master server at 127.12.170.126:38925
I20260926 13:45:06.448549 15862 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:06.448940 15862 heartbeater.cc:511] Master 127.12.170.126:38925 requested a full tablet report, sending...
I20260926 13:45:06.449870 15403 ts_manager.cc:194] Registered new tserver with Master: f218139818d04711bff645d77e61f623 (127.12.170.67:33733)
I20260926 13:45:06.450382 15403 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.67@KRBTEST.COM'} at 127.12.170.67:45637
I20260926 13:45:06.452674 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 13:45:06.463935 15442 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:06.462577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60020 (local address 127.12.170.126:38925)
0926 13:45:06.462676 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:06.462679 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:06.462832 (+ 153us) server_negotiation.cc:407] Connection header received
0926 13:45:06.463001 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:06.463004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:06.463044 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:06.463111 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:06.463796 (+ 685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60020: BlockingRecv error: recv got EOF from 127.0.0.1:60020 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 13:45:06.464844 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 15465
I20260926 13:45:06.469324 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 15599
I20260926 13:45:06.473369 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 15733
I20260926 13:45:06.477536 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 15371
2026-09-26T13:45:06Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1926 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:45:06 dist-test-slave-hc1k krb5kdc[15874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:45:06 dist-test-slave-hc1k krb5kdc[15874](info): set up 2 sockets
Sep 26 13:45:06 dist-test-slave-hc1k krb5kdc[15874](info): commencing operation
krb5kdc: starting...
W20260926 13:45:07.415319 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.919s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:45:07 dist-test-slave-hc1k krb5kdc[15874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:45:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:45:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:45:07.521088 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:43457
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:37385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:43457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:07.623448 15887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:07.623692 15887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:07.623744 15887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:07.627609 15887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:45:07.627669 15887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:07.627687 15887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:07.627702 15887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:45:07.627718 15887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:45:07.632553 15887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:37385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:43457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:43457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:07.633373 15887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:07.634222 15887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:07.641945 15895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:07.641948 15893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:07.641991 15887 server_base.cc:1034] running on GCE node
W20260926 13:45:07.641945 15892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:07.642639 15887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:07.643597 15887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:07.644798 15887 hybrid_clock.cc:648] HybridClock initialized: now 1790430307644782 us; error 30 us; skew 500 ppm
Sep 26 13:45:07 dist-test-slave-hc1k krb5kdc[15874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430307, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:07.647384 15887 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:45:07.648258 15887 webserver.cc:460] Webserver started at http://127.12.170.126:36989/ using document root <none> and password file <none>
I20260926 13:45:07.648732 15887 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:07.648783 15887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:07.648932 15887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:07.650803 15887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "d70ce42d6b2c4df0bb1c8ef230530724"
format_stamp: "Formatted at 2026-09-26 13:45:07 on dist-test-slave-hc1k"
server_key: "425496e5cac754cdb2c4358c25a315c2"
server_key_iv: "4d4b8c5d7e4c3168e1716ea6f76ce8fe"
server_key_version: "encryptionkey@0"
I20260926 13:45:07.651289 15887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "d70ce42d6b2c4df0bb1c8ef230530724"
format_stamp: "Formatted at 2026-09-26 13:45:07 on dist-test-slave-hc1k"
server_key: "425496e5cac754cdb2c4358c25a315c2"
server_key_iv: "4d4b8c5d7e4c3168e1716ea6f76ce8fe"
server_key_version: "encryptionkey@0"
I20260926 13:45:07.654713 15887 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 13:45:07.657055 15902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:07.658236 15887 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:07.658341 15887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "d70ce42d6b2c4df0bb1c8ef230530724"
format_stamp: "Formatted at 2026-09-26 13:45:07 on dist-test-slave-hc1k"
server_key: "425496e5cac754cdb2c4358c25a315c2"
server_key_iv: "4d4b8c5d7e4c3168e1716ea6f76ce8fe"
server_key_version: "encryptionkey@0"
I20260926 13:45:07.658430 15887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:07.676505 15887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:07.679764 15887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:07.679925 15887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:07.686663 15887 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:43457
I20260926 13:45:07.686908 15954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:43457 every 8 connection(s)
I20260926 13:45:07.687573 15887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:45:07.690124 15955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:07.695096 15955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d70ce42d6b2c4df0bb1c8ef230530724: Bootstrap starting.
I20260926 13:45:07.695887 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 15887
I20260926 13:45:07.695990 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:45:07.696269 12969 external_mini_cluster.cc:1428] Setting key 687ebccfe0ed7ee798ee1fa60f893fe8
I20260926 13:45:07.697546 15955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d70ce42d6b2c4df0bb1c8ef230530724: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:07.698199 15955 log.cc:824] T 00000000000000000000000000000000 P d70ce42d6b2c4df0bb1c8ef230530724: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:07.700136 15955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d70ce42d6b2c4df0bb1c8ef230530724: No bootstrap required, opened a new log
I20260926 13:45:07.704015 15955 raft_consensus.cc:348] T 00000000000000000000000000000000 P d70ce42d6b2c4df0bb1c8ef230530724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70ce42d6b2c4df0bb1c8ef230530724" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 43457 } }
I20260926 13:45:07.704293 15955 raft_consensus.cc:374] T 00000000000000000000000000000000 P d70ce42d6b2c4df0bb1c8ef230530724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:07.704352 15955 raft_consensus.cc:729] T 00000000000000000000000000000000 P d70ce42d6b2c4df0bb1c8ef230530724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d70ce42d6b2c4df0bb1c8ef230530724, State: Initialized, Role: FOLLOWER
Sep 26 13:45:07 dist-test-slave-hc1k krb5kdc[15874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:07.704754 15955 consensus_queue.cc:260] T 00000000000000000000000000000000 P d70ce42d6b2c4df0bb1c8ef230530724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70ce42d6b2c4df0bb1c8ef230530724" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 43457 } }
I20260926 13:45:07.704892 15955 raft_consensus.cc:388] T 00000000000000000000000000000000 P d70ce42d6b2c4df0bb1c8ef230530724 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:45:07.704952 15955 raft_consensus.cc:482] T 00000000000000000000000000000000 P d70ce42d6b2c4df0bb1c8ef230530724 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:45:07.705039 15955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d70ce42d6b2c4df0bb1c8ef230530724 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:07.705775 15955 raft_consensus.cc:504] T 00000000000000000000000000000000 P d70ce42d6b2c4df0bb1c8ef230530724 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70ce42d6b2c4df0bb1c8ef230530724" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 43457 } }
I20260926 13:45:07.706061 15955 leader_election.cc:302] T 00000000000000000000000000000000 P d70ce42d6b2c4df0bb1c8ef230530724 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d70ce42d6b2c4df0bb1c8ef230530724; no voters:
I20260926 13:45:07.706305 15955 leader_election.cc:288] T 00000000000000000000000000000000 P d70ce42d6b2c4df0bb1c8ef230530724 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:45:07.706403 15960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d70ce42d6b2c4df0bb1c8ef230530724 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:07.706611 15960 raft_consensus.cc:686] T 00000000000000000000000000000000 P d70ce42d6b2c4df0bb1c8ef230530724 [term 1 LEADER]: Becoming Leader. State: Replica: d70ce42d6b2c4df0bb1c8ef230530724, State: Running, Role: LEADER
I20260926 13:45:07.706913 15960 consensus_queue.cc:237] T 00000000000000000000000000000000 P d70ce42d6b2c4df0bb1c8ef230530724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70ce42d6b2c4df0bb1c8ef230530724" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 43457 } }
I20260926 13:45:07.707494 15955 sys_catalog.cc:564] T 00000000000000000000000000000000 P d70ce42d6b2c4df0bb1c8ef230530724 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:45:07.708429 15962 sys_catalog.cc:455] T 00000000000000000000000000000000 P d70ce42d6b2c4df0bb1c8ef230530724 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d70ce42d6b2c4df0bb1c8ef230530724. Latest consensus state: current_term: 1 leader_uuid: "d70ce42d6b2c4df0bb1c8ef230530724" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70ce42d6b2c4df0bb1c8ef230530724" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 43457 } } }
I20260926 13:45:07.708552 15962 sys_catalog.cc:458] T 00000000000000000000000000000000 P d70ce42d6b2c4df0bb1c8ef230530724 [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:07.708945 15969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:45:07.708891 15961 sys_catalog.cc:455] T 00000000000000000000000000000000 P d70ce42d6b2c4df0bb1c8ef230530724 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d70ce42d6b2c4df0bb1c8ef230530724" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d70ce42d6b2c4df0bb1c8ef230530724" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 43457 } } }
I20260926 13:45:07.709074 15961 sys_catalog.cc:458] T 00000000000000000000000000000000 P d70ce42d6b2c4df0bb1c8ef230530724 [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:07.711004 15958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:07.697232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50808 (local address 127.12.170.126:43457)
0926 13:45:07.697611 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:07.697627 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:07.697668 (+ 41us) server_negotiation.cc:407] Connection header received
0926 13:45:07.698275 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:07.698290 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:07.698669 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:07.699009 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:07.699868 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:07.700868 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:07.702053 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:07.702502 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:07.702617 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:07.705331 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:07.705336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:07.705347 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:07.705373 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:07.707245 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:07.707769 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:07.707776 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:07.707781 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:07.707916 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:07.708320 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:07.708324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:07.708326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:07.708831 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:07.709615 (+ 784us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:07.709795 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 13:45:07.710087 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":114,"threads_started":1}
I20260926 13:45:07.712004 15969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:45:07.713860 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:45:07.717350 15969 catalog_manager.cc:1347] Generated new cluster ID: e902dbc71ad64aa7852ec1431899196a
I20260926 13:45:07.717408 15969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:45:07.734241 15958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:07.729057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50818 (local address 127.12.170.126:43457)
0926 13:45:07.729154 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:07.729159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:07.729410 (+ 251us) server_negotiation.cc:407] Connection header received
0926 13:45:07.729522 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:07.729525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:07.729572 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:07.729648 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:07.730159 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:07.730902 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:07.732048 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:07.732411 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:07.732505 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:07.733311 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:07.733316 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:07.733318 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:45:07.733816 (+ 498us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:07.733847 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:45:07.733866 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:07.733955 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:07.734050 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 13:45:07.734102 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 13:45:07.740005 15969 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:45:07.741046 15969 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:45:07.745713 15969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d70ce42d6b2c4df0bb1c8ef230530724: Generated new TSK 0
I20260926 13:45:07.746368 15969 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:45:07.759938 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 15887
2026-09-26T13:45:07Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1281 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:45:07 dist-test-slave-hc1k krb5kdc[15988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:45:07 dist-test-slave-hc1k krb5kdc[15988](info): set up 2 sockets
Sep 26 13:45:07 dist-test-slave-hc1k krb5kdc[15988](info): commencing operation
krb5kdc: starting...
W20260926 13:45:08.699239 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:45:08 dist-test-slave-hc1k krb5kdc[15988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:45:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:45:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:45:08.806188 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:36969
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:36969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:08.909435 16001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:08.909688 16001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:08.909744 16001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:08.913718 16001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:45:08.913774 16001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:08.913798 16001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:08.913818 16001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:45:08.913844 16001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:45:08.918913 16001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:36969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:36969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:08.919870 16001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:08.920795 16001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:08.928431 16006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:08.928462 16009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:08.928614 16001 server_base.cc:1034] running on GCE node
W20260926 13:45:08.928440 16007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:08.929199 16001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:08.930146 16001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:08.931329 16001 hybrid_clock.cc:648] HybridClock initialized: now 1790430308931304 us; error 36 us; skew 500 ppm
Sep 26 13:45:08 dist-test-slave-hc1k krb5kdc[15988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430308, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:08.933847 16001 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:45:08.934641 16001 webserver.cc:460] Webserver started at http://127.12.170.126:40903/ using document root <none> and password file <none>
I20260926 13:45:08.935153 16001 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:08.935214 16001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:08.935385 16001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:08.937288 16001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "7c7bcc6bf0174d41a93ab7e0f0a6f29b"
format_stamp: "Formatted at 2026-09-26 13:45:08 on dist-test-slave-hc1k"
server_key: "8a9ee586c3041168fc96e7cf17c0160f"
server_key_iv: "5e7844633d3a737f2b4da785a6650a02"
server_key_version: "encryptionkey@0"
I20260926 13:45:08.937799 16001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "7c7bcc6bf0174d41a93ab7e0f0a6f29b"
format_stamp: "Formatted at 2026-09-26 13:45:08 on dist-test-slave-hc1k"
server_key: "8a9ee586c3041168fc96e7cf17c0160f"
server_key_iv: "5e7844633d3a737f2b4da785a6650a02"
server_key_version: "encryptionkey@0"
I20260926 13:45:08.941227 16001 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:45:08.943789 16016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:08.944912 16001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:45:08.945007 16001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "7c7bcc6bf0174d41a93ab7e0f0a6f29b"
format_stamp: "Formatted at 2026-09-26 13:45:08 on dist-test-slave-hc1k"
server_key: "8a9ee586c3041168fc96e7cf17c0160f"
server_key_iv: "5e7844633d3a737f2b4da785a6650a02"
server_key_version: "encryptionkey@0"
I20260926 13:45:08.945099 16001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:08.971529 16001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:08.974314 16001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:08.974467 16001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:08.980687 16001 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:36969
I20260926 13:45:08.980919 16068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:36969 every 8 connection(s)
I20260926 13:45:08.981580 16001 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:45:08.984164 16069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:08.989269 16069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c7bcc6bf0174d41a93ab7e0f0a6f29b: Bootstrap starting.
I20260926 13:45:08.991190 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 16001
I20260926 13:45:08.991282 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:45:08.991515 12969 external_mini_cluster.cc:1428] Setting key a0b4cface92e3b42d6bccde53dea3c25
I20260926 13:45:08.991524 16069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c7bcc6bf0174d41a93ab7e0f0a6f29b: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:08.992322 16069 log.cc:824] T 00000000000000000000000000000000 P 7c7bcc6bf0174d41a93ab7e0f0a6f29b: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:08.994046 16069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c7bcc6bf0174d41a93ab7e0f0a6f29b: No bootstrap required, opened a new log
I20260926 13:45:08.997875 16069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c7bcc6bf0174d41a93ab7e0f0a6f29b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7bcc6bf0174d41a93ab7e0f0a6f29b" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36969 } }
I20260926 13:45:08.998138 16069 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c7bcc6bf0174d41a93ab7e0f0a6f29b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:08.998198 16069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c7bcc6bf0174d41a93ab7e0f0a6f29b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c7bcc6bf0174d41a93ab7e0f0a6f29b, State: Initialized, Role: FOLLOWER
I20260926 13:45:08.998701 16069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c7bcc6bf0174d41a93ab7e0f0a6f29b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7bcc6bf0174d41a93ab7e0f0a6f29b" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36969 } }
I20260926 13:45:08.998863 16069 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c7bcc6bf0174d41a93ab7e0f0a6f29b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:45:08.998931 16069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c7bcc6bf0174d41a93ab7e0f0a6f29b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:45:08.999044 16069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c7bcc6bf0174d41a93ab7e0f0a6f29b [term 0 FOLLOWER]: Advancing to term 1
Sep 26 13:45:08 dist-test-slave-hc1k krb5kdc[15988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:08.999930 16069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c7bcc6bf0174d41a93ab7e0f0a6f29b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7bcc6bf0174d41a93ab7e0f0a6f29b" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36969 } }
I20260926 13:45:09.000295 16069 leader_election.cc:302] T 00000000000000000000000000000000 P 7c7bcc6bf0174d41a93ab7e0f0a6f29b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c7bcc6bf0174d41a93ab7e0f0a6f29b; no voters:
I20260926 13:45:09.000636 16069 leader_election.cc:288] T 00000000000000000000000000000000 P 7c7bcc6bf0174d41a93ab7e0f0a6f29b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:45:09.000715 16074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c7bcc6bf0174d41a93ab7e0f0a6f29b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:09.000882 16074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c7bcc6bf0174d41a93ab7e0f0a6f29b [term 1 LEADER]: Becoming Leader. State: Replica: 7c7bcc6bf0174d41a93ab7e0f0a6f29b, State: Running, Role: LEADER
I20260926 13:45:09.001232 16074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c7bcc6bf0174d41a93ab7e0f0a6f29b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7bcc6bf0174d41a93ab7e0f0a6f29b" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36969 } }
I20260926 13:45:09.001859 16069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c7bcc6bf0174d41a93ab7e0f0a6f29b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:45:09.002921 16075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c7bcc6bf0174d41a93ab7e0f0a6f29b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c7bcc6bf0174d41a93ab7e0f0a6f29b. Latest consensus state: current_term: 1 leader_uuid: "7c7bcc6bf0174d41a93ab7e0f0a6f29b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7bcc6bf0174d41a93ab7e0f0a6f29b" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36969 } } }
I20260926 13:45:09.003041 16075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c7bcc6bf0174d41a93ab7e0f0a6f29b [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:09.003319 16076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c7bcc6bf0174d41a93ab7e0f0a6f29b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c7bcc6bf0174d41a93ab7e0f0a6f29b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7bcc6bf0174d41a93ab7e0f0a6f29b" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36969 } } }
I20260926 13:45:09.003397 16076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c7bcc6bf0174d41a93ab7e0f0a6f29b [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:09.003618 16083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:45:09.005486 16073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:08.992524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35832 (local address 127.12.170.126:36969)
0926 13:45:08.992895 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:08.992918 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:08.992949 (+ 31us) server_negotiation.cc:407] Connection header received
0926 13:45:08.993530 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:08.993546 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:08.993838 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:08.994137 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:08.994978 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:08.995996 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:08.997168 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:08.997607 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:08.997718 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:09.000520 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:09.000526 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:09.000566 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:09.000596 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:09.002437 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:09.002937 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:09.002942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:09.002948 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:09.003081 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:09.003383 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:09.003386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:09.003388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:09.004027 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:09.004227 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:09.004512 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 13:45:09.004794 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":102,"threads_started":1}
I20260926 13:45:09.008080 16083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:45:09.013532 16083 catalog_manager.cc:1347] Generated new cluster ID: 3464e00479a34f1ab3b3a4c65a2b5f69
I20260926 13:45:09.013630 16083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:45:09.029484 16083 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:45:09.030652 16083 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:45:09.041913 16083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c7bcc6bf0174d41a93ab7e0f0a6f29b: Generated new TSK 0
I20260926 13:45:09.042670 16083 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.170.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
I20260926 13:45:09.071998 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:0
--local_ip_for_outbound_sockets=127.12.170.65
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:36969
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:09.176453 16095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:09.176677 16095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:09.176731 16095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:09.180733 16095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:09.180788 16095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:09.180866 16095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:45:09.185860 16095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:36969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:09.186785 16095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:09.187726 16095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:09.196539 16103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:09.196691 16095 server_base.cc:1034] running on GCE node
W20260926 13:45:09.196523 16100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:09.196527 16101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:09.197253 16095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:09.197841 16095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:09.199009 16095 hybrid_clock.cc:648] HybridClock initialized: now 1790430309198994 us; error 30 us; skew 500 ppm
Sep 26 13:45:09 dist-test-slave-hc1k krb5kdc[15988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430309, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:09.201562 16095 init.cc:377] Logged in from keytab as kudu/127.12.170.65@KRBTEST.COM (short username kudu)
I20260926 13:45:09.202389 16095 webserver.cc:460] Webserver started at http://127.12.170.65:40419/ using document root <none> and password file <none>
I20260926 13:45:09.202975 16095 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:09.203044 16095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:09.203233 16095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:09.205214 16095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/data/instance:
uuid: "0a4edb7744c0401baabf5ad013a07e25"
format_stamp: "Formatted at 2026-09-26 13:45:09 on dist-test-slave-hc1k"
server_key: "bac36a3fde28461f0fc3233d16ce84f9"
server_key_iv: "9daff02cdda55bc723bbae5a8d204ecb"
server_key_version: "encryptionkey@0"
I20260926 13:45:09.205719 16095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance:
uuid: "0a4edb7744c0401baabf5ad013a07e25"
format_stamp: "Formatted at 2026-09-26 13:45:09 on dist-test-slave-hc1k"
server_key: "bac36a3fde28461f0fc3233d16ce84f9"
server_key_iv: "9daff02cdda55bc723bbae5a8d204ecb"
server_key_version: "encryptionkey@0"
I20260926 13:45:09.209183 16095 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:45:09.211355 16110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:09.212492 16095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:45:09.212603 16095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "0a4edb7744c0401baabf5ad013a07e25"
format_stamp: "Formatted at 2026-09-26 13:45:09 on dist-test-slave-hc1k"
server_key: "bac36a3fde28461f0fc3233d16ce84f9"
server_key_iv: "9daff02cdda55bc723bbae5a8d204ecb"
server_key_version: "encryptionkey@0"
I20260926 13:45:09.212692 16095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:09.225052 16095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:09.227836 16095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:09.227998 16095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:09.228499 16095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:09.229231 16095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:09.229276 16095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:09.229322 16095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:09.229350 16095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:09.237468 16095 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:36945
I20260926 13:45:09.237514 16223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:36945 every 8 connection(s)
I20260926 13:45:09.238238 16095 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
Sep 26 13:45:09 dist-test-slave-hc1k krb5kdc[15988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430309, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:09.247674 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 16095
I20260926 13:45:09.247792 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance
I20260926 13:45:09.248088 12969 external_mini_cluster.cc:1428] Setting key 90e94015f4026c3525e909173ce4aed3
I20260926 13:45:09.250768 16073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:09.239871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:36953 (local address 127.12.170.126:36969)
0926 13:45:09.240006 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:09.240010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:09.240718 (+ 708us) server_negotiation.cc:407] Connection header received
0926 13:45:09.241952 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:09.241955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:09.242004 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:09.242081 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:09.243103 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:09.243645 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:09.244762 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:09.245002 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:09.245088 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:09.247180 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:09.247183 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:09.247185 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:09.247209 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:09.248737 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:09.249204 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:09.249207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:09.249209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:09.249304 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:09.249676 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:09.249680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:09.249682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:09.249906 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:09.250016 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:09.250473 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 13:45:09.250595 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 13:45:09.251405 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:09.240170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:36969 (local address 127.12.170.65:36953)
0926 13:45:09.240573 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:09.240612 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:09.241769 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:09.242212 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:09.242217 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:09.242516 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:09.242969 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:09.242982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:09.243780 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:09.243783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:09.244647 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:09.244656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:09.245122 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:09.245124 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:09.245251 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:09.245699 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:09.245714 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:09.247035 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:09.248857 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:09.248862 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:09.248872 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:09.249113 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:09.249418 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:09.249421 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:09.249422 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:09.249580 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:09.250007 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:09.250013 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:09.250250 (+ 237us) client_negotiation.cc:769] Sending connection context
0926 13:45:09.250392 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 13:45:09.250608 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":104,"threads_started":1}
I20260926 13:45:09.252655 16224 heartbeater.cc:347] Connected to a master server at 127.12.170.126:36969
I20260926 13:45:09.252847 16224 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:09.253270 16224 heartbeater.cc:511] Master 127.12.170.126:36969 requested a full tablet report, sending...
I20260926 13:45:09.254631 16033 ts_manager.cc:194] Registered new tserver with Master: 0a4edb7744c0401baabf5ad013a07e25 (127.12.170.65:36945)
I20260926 13:45:09.255721 16033 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.65@KRBTEST.COM'} at 127.12.170.65:36953
WARNING: no policy specified for kudu/127.12.170.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
I20260926 13:45:09.271431 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:0
--local_ip_for_outbound_sockets=127.12.170.66
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:36969
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:09.373428 16229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:09.373663 16229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:09.373716 16229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:09.377599 16229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:09.377650 16229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:09.377725 16229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:45:09.382691 16229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:36969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:09.383569 16229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:09.384503 16229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:09.393219 16235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:09.393229 16237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:09.393498 16229 server_base.cc:1034] running on GCE node
W20260926 13:45:09.393311 16234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:09.393987 16229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:09.394551 16229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:09.395709 16229 hybrid_clock.cc:648] HybridClock initialized: now 1790430309395693 us; error 30 us; skew 500 ppm
Sep 26 13:45:09 dist-test-slave-hc1k krb5kdc[15988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430309, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:09.398280 16229 init.cc:377] Logged in from keytab as kudu/127.12.170.66@KRBTEST.COM (short username kudu)
I20260926 13:45:09.399075 16229 webserver.cc:460] Webserver started at http://127.12.170.66:36417/ using document root <none> and password file <none>
I20260926 13:45:09.399565 16229 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:09.399628 16229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:09.399820 16229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:09.401702 16229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/data/instance:
uuid: "0c0cf96febb24fa199fea4bf4594fddf"
format_stamp: "Formatted at 2026-09-26 13:45:09 on dist-test-slave-hc1k"
server_key: "08d396edfd0b181b22198ee339bb083f"
server_key_iv: "03d668e227549023d661d7a10c92c099"
server_key_version: "encryptionkey@0"
I20260926 13:45:09.402266 16229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance:
uuid: "0c0cf96febb24fa199fea4bf4594fddf"
format_stamp: "Formatted at 2026-09-26 13:45:09 on dist-test-slave-hc1k"
server_key: "08d396edfd0b181b22198ee339bb083f"
server_key_iv: "03d668e227549023d661d7a10c92c099"
server_key_version: "encryptionkey@0"
I20260926 13:45:09.405736 16229 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:45:09.407898 16244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:09.409011 16229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 13:45:09.409109 16229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "0c0cf96febb24fa199fea4bf4594fddf"
format_stamp: "Formatted at 2026-09-26 13:45:09 on dist-test-slave-hc1k"
server_key: "08d396edfd0b181b22198ee339bb083f"
server_key_iv: "03d668e227549023d661d7a10c92c099"
server_key_version: "encryptionkey@0"
I20260926 13:45:09.409193 16229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:09.421171 16229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:09.423915 16229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:09.424078 16229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:09.424559 16229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:09.425297 16229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:09.425344 16229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:09.425393 16229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:09.425422 16229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:09.433481 16229 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:42213
I20260926 13:45:09.433519 16357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:42213 every 8 connection(s)
I20260926 13:45:09.434262 16229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:45:09.436148 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 16229
I20260926 13:45:09.436257 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance
I20260926 13:45:09.436517 12969 external_mini_cluster.cc:1428] Setting key 22f9bcc7d72132310833a4c913912215
Sep 26 13:45:09 dist-test-slave-hc1k krb5kdc[15988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430309, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.170.67@KRBTEST.COM; defaulting to no policy
I20260926 13:45:09.447952 16073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:09.435886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:60257 (local address 127.12.170.126:36969)
0926 13:45:09.436053 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:09.436058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:09.436880 (+ 822us) server_negotiation.cc:407] Connection header received
0926 13:45:09.438345 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:09.438348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:09.438396 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:09.438477 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:09.439543 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:09.440230 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:09.441847 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:09.442072 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:09.442153 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:09.444833 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:09.444836 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:09.444838 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:09.444861 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:09.446142 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:09.446588 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:09.446591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:09.446592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:09.446661 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:09.447016 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:09.447018 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:09.447020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:09.447188 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:09.447263 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:09.447712 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 13:45:09.447815 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.67@KRBTEST.COM" created.
I20260926 13:45:09.448874 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:09.436264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:36969 (local address 127.12.170.66:60257)
0926 13:45:09.436733 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:09.436784 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:09.438143 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:09.438606 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:09.438612 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:09.438954 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:09.439395 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:09.439409 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:09.440413 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:09.440416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:09.441727 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:09.441740 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:09.442224 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:09.442229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:09.442438 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:09.442979 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:09.443002 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:09.444668 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:09.446244 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:09.446249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:09.446259 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:09.446509 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:09.446763 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:09.446766 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:09.446768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:09.446936 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:09.447266 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:09.447271 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:09.447486 (+ 215us) client_negotiation.cc:769] Sending connection context
0926 13:45:09.447677 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 13:45:09.447964 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":108,"threads_started":1}
I20260926 13:45:09.449903 16358 heartbeater.cc:347] Connected to a master server at 127.12.170.126:36969
I20260926 13:45:09.450119 16358 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:09.450558 16358 heartbeater.cc:511] Master 127.12.170.126:36969 requested a full tablet report, sending...
I20260926 13:45:09.451581 16033 ts_manager.cc:194] Registered new tserver with Master: 0c0cf96febb24fa199fea4bf4594fddf (127.12.170.66:42213)
I20260926 13:45:09.452150 16033 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.66@KRBTEST.COM'} at 127.12.170.66:60257
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
I20260926 13:45:09.462397 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:0
--local_ip_for_outbound_sockets=127.12.170.67
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:36969
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:09.566883 16363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:09.567134 16363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:09.567188 16363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:09.571084 16363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:09.571141 16363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:09.571218 16363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:45:09.576258 16363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:36969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:09.577193 16363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:09.578141 16363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:09.587350 16369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:09.587358 16368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:09.587502 16363 server_base.cc:1034] running on GCE node
W20260926 13:45:09.587363 16371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:09.587963 16363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:09.588531 16363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:09.589686 16363 hybrid_clock.cc:648] HybridClock initialized: now 1790430309589667 us; error 32 us; skew 500 ppm
Sep 26 13:45:09 dist-test-slave-hc1k krb5kdc[15988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430309, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:09.592253 16363 init.cc:377] Logged in from keytab as kudu/127.12.170.67@KRBTEST.COM (short username kudu)
I20260926 13:45:09.593029 16363 webserver.cc:460] Webserver started at http://127.12.170.67:36577/ using document root <none> and password file <none>
I20260926 13:45:09.593530 16363 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:09.593596 16363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:09.593776 16363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:09.595716 16363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/data/instance:
uuid: "85e2af803ddf43f3a11c0bab2eb04458"
format_stamp: "Formatted at 2026-09-26 13:45:09 on dist-test-slave-hc1k"
server_key: "1ce34dd9f79f77819ddaf19ca339db0c"
server_key_iv: "d3648d62af19edecb76da5c10a143417"
server_key_version: "encryptionkey@0"
I20260926 13:45:09.596258 16363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance:
uuid: "85e2af803ddf43f3a11c0bab2eb04458"
format_stamp: "Formatted at 2026-09-26 13:45:09 on dist-test-slave-hc1k"
server_key: "1ce34dd9f79f77819ddaf19ca339db0c"
server_key_iv: "d3648d62af19edecb76da5c10a143417"
server_key_version: "encryptionkey@0"
I20260926 13:45:09.599673 16363 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260926 13:45:09.602007 16378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:09.603241 16363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:09.603339 16363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "85e2af803ddf43f3a11c0bab2eb04458"
format_stamp: "Formatted at 2026-09-26 13:45:09 on dist-test-slave-hc1k"
server_key: "1ce34dd9f79f77819ddaf19ca339db0c"
server_key_iv: "d3648d62af19edecb76da5c10a143417"
server_key_version: "encryptionkey@0"
I20260926 13:45:09.603426 16363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:09.630852 16363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:09.633574 16363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:09.633738 16363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:09.634276 16363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:09.635051 16363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:09.635097 16363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:09.635157 16363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:09.635185 16363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:09.643200 16363 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:40167
I20260926 13:45:09.643237 16491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:40167 every 8 connection(s)
I20260926 13:45:09.644027 16363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:45:09.648034 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 16363
I20260926 13:45:09.648133 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance
I20260926 13:45:09.648396 12969 external_mini_cluster.cc:1428] Setting key 36c967f3ddb55dabb7f0dbb68913f126
Sep 26 13:45:09 dist-test-slave-hc1k krb5kdc[15988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430309, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:09.656491 16073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:09.645630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:33269 (local address 127.12.170.126:36969)
0926 13:45:09.645748 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:09.645752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:09.646553 (+ 801us) server_negotiation.cc:407] Connection header received
0926 13:45:09.647898 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:09.647901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:09.647961 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:09.648040 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:09.649359 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:09.649955 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:09.651089 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:09.651299 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:09.651382 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:09.653560 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:09.653565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:09.653567 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:09.653589 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:09.654654 (+ 1065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:09.655113 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:09.655117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:09.655119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:09.655190 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:09.655490 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:09.655494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:09.655495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:09.655686 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:09.655773 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:09.656212 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 13:45:09.656328 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:45:09.657181 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:09.646000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:36969 (local address 127.12.170.67:33269)
0926 13:45:09.646402 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:09.646446 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:09.647672 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:09.648238 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:09.648246 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:09.648634 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:09.649227 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:09.649248 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:09.650067 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:09.650070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:09.650975 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:09.650984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:09.651432 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:09.651437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:09.651600 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:09.652137 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:09.652152 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:09.653419 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:09.654751 (+ 1332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:09.654756 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:09.654770 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:09.655027 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:09.655267 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:09.655270 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:09.655272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:09.655406 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:09.655780 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:09.655786 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:09.656006 (+ 220us) client_negotiation.cc:769] Sending connection context
0926 13:45:09.656159 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 13:45:09.656443 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":95,"threads_started":1}
I20260926 13:45:09.658125 16492 heartbeater.cc:347] Connected to a master server at 127.12.170.126:36969
I20260926 13:45:09.658315 16492 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:09.658708 16492 heartbeater.cc:511] Master 127.12.170.126:36969 requested a full tablet report, sending...
I20260926 13:45:09.659693 16033 ts_manager.cc:194] Registered new tserver with Master: 85e2af803ddf43f3a11c0bab2eb04458 (127.12.170.67:40167)
I20260926 13:45:09.660223 16033 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.67@KRBTEST.COM'} at 127.12.170.67:33269
I20260926 13:45:09.661828 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 13:45:09 dist-test-slave-hc1k krb5kdc[15988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430309, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 13:45:09 dist-test-slave-hc1k krb5kdc[15988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430309, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:09.685513 16073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:09.677444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35836 (local address 127.12.170.126:36969)
0926 13:45:09.677601 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:09.677606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:09.677734 (+ 128us) server_negotiation.cc:407] Connection header received
0926 13:45:09.677901 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:09.677905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:09.677952 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:09.678022 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:09.678765 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:09.679314 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:09.680193 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:09.680416 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:09.680506 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:09.682831 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:09.682834 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:09.682836 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:09.682858 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:09.684056 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:09.684406 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:09.684409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:09.684411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:09.684482 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:09.684729 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:09.684732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:09.684734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:09.684938 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:09.685050 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:09.685256 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 13:45:09.685357 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 13:45:09.685853 16033 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35836
I20260926 13:45:09.686645 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 16095
I20260926 13:45:09.691398 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 16229
I20260926 13:45:09.695771 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 16363
I20260926 13:45:09.700099 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 16001
2026-09-26T13:45:09Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1941 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:45:09 dist-test-slave-hc1k krb5kdc[16504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:45:09 dist-test-slave-hc1k krb5kdc[16504](info): set up 2 sockets
Sep 26 13:45:09 dist-test-slave-hc1k krb5kdc[16504](info): commencing operation
krb5kdc: starting...
W20260926 13:45:10.643275 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:45:10 dist-test-slave-hc1k krb5kdc[16504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:45:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:45:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:45:10.750134 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:46545
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:39409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:46545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:10.850689 16517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:10.851006 16517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:10.851082 16517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:10.856484 16517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:45:10.856539 16517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:10.856559 16517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:10.856573 16517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:45:10.856592 16517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:45:10.861608 16517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:39409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:46545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:46545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:10.862465 16517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:10.863327 16517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:10.870941 16525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:10.870904 16522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:10.870893 16517 server_base.cc:1034] running on GCE node
W20260926 13:45:10.870874 16523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:10.871563 16517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:10.872562 16517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:10.873708 16517 hybrid_clock.cc:648] HybridClock initialized: now 1790430310873691 us; error 36 us; skew 500 ppm
Sep 26 13:45:10 dist-test-slave-hc1k krb5kdc[16504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430310, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:10.876087 16517 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:45:10.876868 16517 webserver.cc:460] Webserver started at http://127.12.170.126:45089/ using document root <none> and password file <none>
I20260926 13:45:10.877350 16517 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:10.877410 16517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:10.877591 16517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:10.879418 16517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "6ca604adf7a44dd2a1c539f3fccdc32d"
format_stamp: "Formatted at 2026-09-26 13:45:10 on dist-test-slave-hc1k"
server_key: "afc36a389784c58c1d17c43b4529dbed"
server_key_iv: "812af27e209acc6838ee7db623d1c960"
server_key_version: "encryptionkey@0"
I20260926 13:45:10.879959 16517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "6ca604adf7a44dd2a1c539f3fccdc32d"
format_stamp: "Formatted at 2026-09-26 13:45:10 on dist-test-slave-hc1k"
server_key: "afc36a389784c58c1d17c43b4529dbed"
server_key_iv: "812af27e209acc6838ee7db623d1c960"
server_key_version: "encryptionkey@0"
I20260926 13:45:10.883235 16517 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:45:10.885448 16532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:10.886504 16517 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:10.886618 16517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "6ca604adf7a44dd2a1c539f3fccdc32d"
format_stamp: "Formatted at 2026-09-26 13:45:10 on dist-test-slave-hc1k"
server_key: "afc36a389784c58c1d17c43b4529dbed"
server_key_iv: "812af27e209acc6838ee7db623d1c960"
server_key_version: "encryptionkey@0"
I20260926 13:45:10.886701 16517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:10.919028 16517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:10.921898 16517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:10.922066 16517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:10.928622 16517 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:46545
I20260926 13:45:10.928643 16584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:46545 every 8 connection(s)
I20260926 13:45:10.929477 16517 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:45:10.932214 16585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:10.934862 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 16517
I20260926 13:45:10.934994 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:45:10.935310 12969 external_mini_cluster.cc:1428] Setting key 85e94012bdaeefa6373dee116f03f1c7
I20260926 13:45:10.938000 16585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ca604adf7a44dd2a1c539f3fccdc32d: Bootstrap starting.
I20260926 13:45:10.940124 16585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ca604adf7a44dd2a1c539f3fccdc32d: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:10.940801 16585 log.cc:824] T 00000000000000000000000000000000 P 6ca604adf7a44dd2a1c539f3fccdc32d: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:10.942600 16585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ca604adf7a44dd2a1c539f3fccdc32d: No bootstrap required, opened a new log
Sep 26 13:45:10 dist-test-slave-hc1k krb5kdc[16504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:10.945755 16585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ca604adf7a44dd2a1c539f3fccdc32d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca604adf7a44dd2a1c539f3fccdc32d" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 46545 } }
I20260926 13:45:10.945978 16585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ca604adf7a44dd2a1c539f3fccdc32d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:10.946017 16585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ca604adf7a44dd2a1c539f3fccdc32d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ca604adf7a44dd2a1c539f3fccdc32d, State: Initialized, Role: FOLLOWER
I20260926 13:45:10.946467 16585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ca604adf7a44dd2a1c539f3fccdc32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca604adf7a44dd2a1c539f3fccdc32d" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 46545 } }
I20260926 13:45:10.946627 16585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ca604adf7a44dd2a1c539f3fccdc32d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:45:10.946702 16585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ca604adf7a44dd2a1c539f3fccdc32d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:45:10.946801 16585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ca604adf7a44dd2a1c539f3fccdc32d [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:10.947835 16585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ca604adf7a44dd2a1c539f3fccdc32d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca604adf7a44dd2a1c539f3fccdc32d" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 46545 } }
I20260926 13:45:10.948185 16585 leader_election.cc:302] T 00000000000000000000000000000000 P 6ca604adf7a44dd2a1c539f3fccdc32d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ca604adf7a44dd2a1c539f3fccdc32d; no voters:
I20260926 13:45:10.948489 16585 leader_election.cc:288] T 00000000000000000000000000000000 P 6ca604adf7a44dd2a1c539f3fccdc32d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:45:10.948581 16590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ca604adf7a44dd2a1c539f3fccdc32d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:10.948774 16590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ca604adf7a44dd2a1c539f3fccdc32d [term 1 LEADER]: Becoming Leader. State: Replica: 6ca604adf7a44dd2a1c539f3fccdc32d, State: Running, Role: LEADER
I20260926 13:45:10.948956 16588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:10.936616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57164 (local address 127.12.170.126:46545)
0926 13:45:10.936990 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:10.937004 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:10.937037 (+ 33us) server_negotiation.cc:407] Connection header received
0926 13:45:10.937616 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:10.937632 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:10.937937 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:10.938243 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:10.938993 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:10.939996 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:10.940835 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:10.941311 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:10.941454 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:10.943913 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:10.943918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:10.943931 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:10.943955 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:10.945962 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:10.946377 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:10.946383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:10.946389 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:10.946539 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:10.946815 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:10.946819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:10.946821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:10.947305 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:10.947525 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:10.947733 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 13:45:10.948022 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":103,"threads_started":1}
I20260926 13:45:10.949095 16590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ca604adf7a44dd2a1c539f3fccdc32d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca604adf7a44dd2a1c539f3fccdc32d" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 46545 } }
I20260926 13:45:10.949692 16585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ca604adf7a44dd2a1c539f3fccdc32d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:45:10.950453 16592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ca604adf7a44dd2a1c539f3fccdc32d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ca604adf7a44dd2a1c539f3fccdc32d. Latest consensus state: current_term: 1 leader_uuid: "6ca604adf7a44dd2a1c539f3fccdc32d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca604adf7a44dd2a1c539f3fccdc32d" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 46545 } } }
I20260926 13:45:10.950582 16592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ca604adf7a44dd2a1c539f3fccdc32d [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:10.950886 16591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ca604adf7a44dd2a1c539f3fccdc32d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ca604adf7a44dd2a1c539f3fccdc32d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ca604adf7a44dd2a1c539f3fccdc32d" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 46545 } } }
I20260926 13:45:10.950958 16591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ca604adf7a44dd2a1c539f3fccdc32d [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:10.951565 16594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:45:10.954552 16594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:45:10.959909 16594 catalog_manager.cc:1347] Generated new cluster ID: e06a7d544ac745bca9c1a4241e0e3c95
I20260926 13:45:10.959965 16594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:45:10.972641 16594 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:45:10.973523 16594 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:45:10.978022 16594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ca604adf7a44dd2a1c539f3fccdc32d: Generated new TSK 0
I20260926 13:45:10.978636 16594 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.170.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
I20260926 13:45:11.011487 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:0
--local_ip_for_outbound_sockets=127.12.170.65
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:46545
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:39409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:11.115278 16611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:11.115509 16611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:11.115571 16611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:11.120357 16611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:11.120410 16611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:11.120488 16611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:45:11.125453 16611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:39409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:46545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:11.126344 16611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:11.127251 16611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:11.135767 16616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:11.135798 16611 server_base.cc:1034] running on GCE node
W20260926 13:45:11.135823 16617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:11.135761 16619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:11.136384 16611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:11.136933 16611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:11.138079 16611 hybrid_clock.cc:648] HybridClock initialized: now 1790430311138056 us; error 36 us; skew 500 ppm
Sep 26 13:45:11 dist-test-slave-hc1k krb5kdc[16504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430311, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:11.140630 16611 init.cc:377] Logged in from keytab as kudu/127.12.170.65@KRBTEST.COM (short username kudu)
I20260926 13:45:11.141402 16611 webserver.cc:460] Webserver started at http://127.12.170.65:35487/ using document root <none> and password file <none>
I20260926 13:45:11.141901 16611 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:11.141963 16611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:11.142144 16611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:11.144024 16611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-0/data/instance:
uuid: "c79e537fc50e440f8da03b5594bd19cf"
format_stamp: "Formatted at 2026-09-26 13:45:11 on dist-test-slave-hc1k"
server_key: "4eb274c68052f7ae8ddd939295b5d8fb"
server_key_iv: "3810e4969a5e25c7f448331b71393fff"
server_key_version: "encryptionkey@0"
I20260926 13:45:11.144533 16611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance:
uuid: "c79e537fc50e440f8da03b5594bd19cf"
format_stamp: "Formatted at 2026-09-26 13:45:11 on dist-test-slave-hc1k"
server_key: "4eb274c68052f7ae8ddd939295b5d8fb"
server_key_iv: "3810e4969a5e25c7f448331b71393fff"
server_key_version: "encryptionkey@0"
I20260926 13:45:11.147894 16611 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:45:11.150084 16626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:11.151216 16611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:11.151309 16611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "c79e537fc50e440f8da03b5594bd19cf"
format_stamp: "Formatted at 2026-09-26 13:45:11 on dist-test-slave-hc1k"
server_key: "4eb274c68052f7ae8ddd939295b5d8fb"
server_key_iv: "3810e4969a5e25c7f448331b71393fff"
server_key_version: "encryptionkey@0"
I20260926 13:45:11.151394 16611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:11.172924 16611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:11.175526 16611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:11.175706 16611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:11.176237 16611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:11.176975 16611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:11.177021 16611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:11.177080 16611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:11.177107 16611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:11.185004 16611 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:40013
I20260926 13:45:11.185053 16739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:40013 every 8 connection(s)
I20260926 13:45:11.185830 16611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:45:11.186893 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 16611
I20260926 13:45:11.186995 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance
I20260926 13:45:11.187248 12969 external_mini_cluster.cc:1428] Setting key 64985eecaa78dd84a7f7b9b8bf9ff2d1
Sep 26 13:45:11 dist-test-slave-hc1k krb5kdc[16504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430311, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.170.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.66@KRBTEST.COM" created.
I20260926 13:45:11.199812 16588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:11.187549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:39125 (local address 127.12.170.126:46545)
0926 13:45:11.187712 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:11.187716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:11.188585 (+ 869us) server_negotiation.cc:407] Connection header received
0926 13:45:11.189998 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:11.190002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:11.190058 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:11.190138 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:11.191322 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:11.192125 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:11.193329 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:11.193707 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:11.193839 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:11.196461 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:11.196464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:11.196466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:11.196488 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:11.197775 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:11.198243 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:11.198245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:11.198246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:11.198316 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:11.198634 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:11.198636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:11.198638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:11.198809 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:11.198883 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:11.199481 (+ 598us) server_negotiation.cc:299] Negotiation successful
0926 13:45:11.199613 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 13:45:11.200528 16742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:11.187956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:46545 (local address 127.12.170.65:39125)
0926 13:45:11.188423 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:11.188484 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:11.189794 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:11.190296 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:11.190302 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:11.190676 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:11.191180 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:11.191196 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:11.192256 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:11.192260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:11.193201 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:11.193211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:11.193860 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:11.193863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:11.193999 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:11.194580 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:11.194595 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:11.196280 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:11.197889 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:11.197896 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:11.197907 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:11.198162 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:11.198398 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:11.198400 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:11.198401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:11.198541 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:11.198953 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:11.198958 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:11.199219 (+ 261us) client_negotiation.cc:769] Sending connection context
0926 13:45:11.199414 (+ 195us) client_negotiation.cc:240] Negotiation successful
0926 13:45:11.199747 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":118,"threads_started":1}
I20260926 13:45:11.201671 16740 heartbeater.cc:347] Connected to a master server at 127.12.170.126:46545
I20260926 13:45:11.201880 16740 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:11.202315 16740 heartbeater.cc:511] Master 127.12.170.126:46545 requested a full tablet report, sending...
I20260926 13:45:11.203761 16549 ts_manager.cc:194] Registered new tserver with Master: c79e537fc50e440f8da03b5594bd19cf (127.12.170.65:40013)
I20260926 13:45:11.204850 16549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.65@KRBTEST.COM'} at 127.12.170.65:39125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
I20260926 13:45:11.212494 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:0
--local_ip_for_outbound_sockets=127.12.170.66
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:46545
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:39409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:11.314795 16745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:11.315014 16745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:11.315068 16745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:11.318926 16745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:11.318976 16745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:11.319051 16745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:45:11.324206 16745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:39409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:46545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:11.325093 16745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:11.325992 16745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:11.334555 16753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:11.334677 16745 server_base.cc:1034] running on GCE node
W20260926 13:45:11.334630 16750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:11.334570 16751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:11.335186 16745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:11.335788 16745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:11.336939 16745 hybrid_clock.cc:648] HybridClock initialized: now 1790430311336923 us; error 29 us; skew 500 ppm
Sep 26 13:45:11 dist-test-slave-hc1k krb5kdc[16504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430311, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:11.339557 16745 init.cc:377] Logged in from keytab as kudu/127.12.170.66@KRBTEST.COM (short username kudu)
I20260926 13:45:11.340330 16745 webserver.cc:460] Webserver started at http://127.12.170.66:32817/ using document root <none> and password file <none>
I20260926 13:45:11.340799 16745 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:11.340852 16745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:11.341005 16745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:11.342913 16745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-1/data/instance:
uuid: "026a5a3b7da84690ad91b55d1e4b49f9"
format_stamp: "Formatted at 2026-09-26 13:45:11 on dist-test-slave-hc1k"
server_key: "f78c284402be7c008c818276cbfe0983"
server_key_iv: "2bc11514aa2d429154f286907dba790d"
server_key_version: "encryptionkey@0"
I20260926 13:45:11.343375 16745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance:
uuid: "026a5a3b7da84690ad91b55d1e4b49f9"
format_stamp: "Formatted at 2026-09-26 13:45:11 on dist-test-slave-hc1k"
server_key: "f78c284402be7c008c818276cbfe0983"
server_key_iv: "2bc11514aa2d429154f286907dba790d"
server_key_version: "encryptionkey@0"
I20260926 13:45:11.346832 16745 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 13:45:11.349050 16760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:11.350136 16745 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:11.350250 16745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "026a5a3b7da84690ad91b55d1e4b49f9"
format_stamp: "Formatted at 2026-09-26 13:45:11 on dist-test-slave-hc1k"
server_key: "f78c284402be7c008c818276cbfe0983"
server_key_iv: "2bc11514aa2d429154f286907dba790d"
server_key_version: "encryptionkey@0"
I20260926 13:45:11.350337 16745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:11.369545 16745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:11.372433 16745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:11.372572 16745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:11.373054 16745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:11.373785 16745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:11.373828 16745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:11.373864 16745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:11.373884 16745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:11.381829 16745 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:39751
I20260926 13:45:11.381943 16873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:39751 every 8 connection(s)
I20260926 13:45:11.382637 16745 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:45:11.387565 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 16745
I20260926 13:45:11.387701 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance
I20260926 13:45:11.388041 12969 external_mini_cluster.cc:1428] Setting key dda6026e2894562aa6aba85ce1d423a9
Sep 26 13:45:11 dist-test-slave-hc1k krb5kdc[16504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430311, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:11.395828 16588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:11.384194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:53301 (local address 127.12.170.126:46545)
0926 13:45:11.384312 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:11.384316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:11.385099 (+ 783us) server_negotiation.cc:407] Connection header received
0926 13:45:11.386406 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:11.386409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:11.386453 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:11.386529 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:11.387604 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:11.388356 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:11.389809 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:11.390108 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:11.390206 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:11.392525 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:11.392528 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:11.392530 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:11.392551 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:11.393876 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:11.394426 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:11.394428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:11.394430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:11.394493 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:11.394832 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:11.394834 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:11.394835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:11.394985 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:11.395049 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:11.395589 (+ 540us) server_negotiation.cc:299] Negotiation successful
0926 13:45:11.395710 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 13:45:11.396910 16876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:11.384532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:46545 (local address 127.12.170.66:53301)
0926 13:45:11.384931 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:11.384970 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:11.386216 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:11.386673 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:11.386679 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:11.386994 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:11.387467 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:11.387481 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:11.388488 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:11.388492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:11.389681 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:11.389691 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:11.390243 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:11.390246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:11.390386 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:11.390916 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:11.390936 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:11.392377 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:11.393986 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:11.393992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:11.394007 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:11.394333 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:11.394568 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:11.394571 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:11.394573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:11.394757 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:11.395073 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:11.395080 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:11.395362 (+ 282us) client_negotiation.cc:769] Sending connection context
0926 13:45:11.395562 (+ 200us) client_negotiation.cc:240] Negotiation successful
0926 13:45:11.395924 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.12.170.67@KRBTEST.COM; defaulting to no policy
I20260926 13:45:11.397966 16874 heartbeater.cc:347] Connected to a master server at 127.12.170.126:46545
I20260926 13:45:11.398192 16874 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:11.398622 16874 heartbeater.cc:511] Master 127.12.170.126:46545 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.67@KRBTEST.COM" created.
I20260926 13:45:11.399708 16549 ts_manager.cc:194] Registered new tserver with Master: 026a5a3b7da84690ad91b55d1e4b49f9 (127.12.170.66:39751)
I20260926 13:45:11.400372 16549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.66@KRBTEST.COM'} at 127.12.170.66:53301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
I20260926 13:45:11.412595 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:0
--local_ip_for_outbound_sockets=127.12.170.67
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:46545
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:39409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:11.516111 16879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:11.516322 16879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:11.516359 16879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:11.520629 16879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:11.520676 16879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:11.520730 16879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:45:11.525570 16879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:39409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:46545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:11.526360 16879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:11.527182 16879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:45:11.535759 16879 server_base.cc:1034] running on GCE node
W20260926 13:45:11.535799 16887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:11.535787 16884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:11.535778 16885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:11.536315 16879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:11.536823 16879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:11.537973 16879 hybrid_clock.cc:648] HybridClock initialized: now 1790430311537958 us; error 26 us; skew 500 ppm
Sep 26 13:45:11 dist-test-slave-hc1k krb5kdc[16504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430311, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:11.543833 16879 init.cc:377] Logged in from keytab as kudu/127.12.170.67@KRBTEST.COM (short username kudu)
I20260926 13:45:11.544607 16879 webserver.cc:460] Webserver started at http://127.12.170.67:39743/ using document root <none> and password file <none>
I20260926 13:45:11.545085 16879 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:11.545143 16879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:11.545307 16879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:11.547194 16879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-2/data/instance:
uuid: "939100c3ab884b80b1707ed51ebe98b1"
format_stamp: "Formatted at 2026-09-26 13:45:11 on dist-test-slave-hc1k"
server_key: "970331b2df6c89b318b96cbfd1332c83"
server_key_iv: "acf2e3f27dba07c952b433f69a30f0ab"
server_key_version: "encryptionkey@0"
I20260926 13:45:11.547729 16879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance:
uuid: "939100c3ab884b80b1707ed51ebe98b1"
format_stamp: "Formatted at 2026-09-26 13:45:11 on dist-test-slave-hc1k"
server_key: "970331b2df6c89b318b96cbfd1332c83"
server_key_iv: "acf2e3f27dba07c952b433f69a30f0ab"
server_key_version: "encryptionkey@0"
I20260926 13:45:11.551079 16879 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:45:11.553417 16894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:11.554620 16879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 13:45:11.554718 16879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "939100c3ab884b80b1707ed51ebe98b1"
format_stamp: "Formatted at 2026-09-26 13:45:11 on dist-test-slave-hc1k"
server_key: "970331b2df6c89b318b96cbfd1332c83"
server_key_iv: "acf2e3f27dba07c952b433f69a30f0ab"
server_key_version: "encryptionkey@0"
I20260926 13:45:11.554797 16879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:11.574088 16879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:11.576786 16879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:11.576931 16879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:11.577370 16879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:11.578097 16879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:11.578140 16879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:11.578173 16879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:11.578192 16879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:11.585978 16879 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:41547
I20260926 13:45:11.586006 17007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:41547 every 8 connection(s)
I20260926 13:45:11.586727 16879 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:45:11.587743 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 16879
I20260926 13:45:11.587828 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance
I20260926 13:45:11.588078 12969 external_mini_cluster.cc:1428] Setting key bd291b98f546a39932934695fb1906a9
Sep 26 13:45:11 dist-test-slave-hc1k krb5kdc[16504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430311, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:11.599037 16588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:11.588371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:57579 (local address 127.12.170.126:46545)
0926 13:45:11.588523 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:11.588528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:11.589342 (+ 814us) server_negotiation.cc:407] Connection header received
0926 13:45:11.590722 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:11.590724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:11.590767 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:11.590828 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:11.591899 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:11.592422 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:11.593444 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:11.593658 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:11.593727 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:11.595986 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:11.595989 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:11.595991 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:11.596014 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:11.597308 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:11.597760 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:11.597762 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:11.597764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:11.597835 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:11.598158 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:11.598160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:11.598161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:11.598319 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:11.598386 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:11.598803 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 13:45:11.598890 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 13:45:11.599735 17010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:11.588755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:46545 (local address 127.12.170.67:57579)
0926 13:45:11.589207 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:11.589255 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:11.590559 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:11.590950 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:11.590956 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:11.591287 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:11.591782 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:11.591796 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:11.592509 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:11.592511 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:11.593330 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:11.593338 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:11.593764 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:11.593767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:11.593900 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:11.594354 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:11.594389 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:11.595827 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:11.597416 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:11.597422 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:11.597433 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:11.597676 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:11.597936 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:11.597939 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:11.597940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:11.598082 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:11.598393 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:11.598398 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:11.598603 (+ 205us) client_negotiation.cc:769] Sending connection context
0926 13:45:11.598749 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 13:45:11.598970 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":126,"threads_started":1}
I20260926 13:45:11.600769 17008 heartbeater.cc:347] Connected to a master server at 127.12.170.126:46545
I20260926 13:45:11.601012 17008 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:11.601461 17008 heartbeater.cc:511] Master 127.12.170.126:46545 requested a full tablet report, sending...
I20260926 13:45:11.602391 16549 ts_manager.cc:194] Registered new tserver with Master: 939100c3ab884b80b1707ed51ebe98b1 (127.12.170.67:41547)
I20260926 13:45:11.602932 16549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.67@KRBTEST.COM'} at 127.12.170.67:57579
I20260926 13:45:11.613114 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 13:45:11 dist-test-slave-hc1k krb5kdc[16504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 13:45:11 dist-test-slave-hc1k krb5kdc[16504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.65@KRBTEST.COM
I20260926 13:45:11.636802 17015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:11.628567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.12.170.65:40013)
0926 13:45:11.628890 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:11.628901 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:11.628923 (+ 22us) server_negotiation.cc:407] Connection header received
0926 13:45:11.628968 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:11.628972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:11.629143 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:11.629252 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:11.629811 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:11.630555 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:11.631413 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:11.631801 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:11.631882 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:11.634035 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:11.634043 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:11.634054 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:11.634075 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:11.635362 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:11.635749 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:11.635753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:11.635754 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:11.635824 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:11.636035 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:11.636038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:11.636039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:11.636250 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:11.636395 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:11.636544 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 13:45:11.636663 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":124,"threads_started":1}
Sep 26 13:45:11 dist-test-slave-hc1k krb5kdc[16504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:11.646477 16588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:11.639347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57168 (local address 127.12.170.126:46545)
0926 13:45:11.639491 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:11.639494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:11.639506 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:45:11.639614 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:11.639616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:11.639677 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:11.639742 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:11.640280 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:11.640763 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:11.641491 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:11.641697 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:11.641762 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:11.643888 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:11.643892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:11.643895 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:11.643917 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:11.645038 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:11.645391 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:11.645395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:11.645397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:11.645482 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:11.645710 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:11.645713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:11.645715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:11.645905 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:11.646020 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:11.646229 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 13:45:11.646331 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 13:45:11.646893 16549 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57168
I20260926 13:45:11.647449 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 16611
I20260926 13:45:11.652493 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 16745
I20260926 13:45:11.656338 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 16879
I20260926 13:45:11.660378 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 16517
2026-09-26T13:45:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1960 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:45:11 dist-test-slave-hc1k krb5kdc[17020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:45:11 dist-test-slave-hc1k krb5kdc[17020](info): set up 2 sockets
Sep 26 13:45:11 dist-test-slave-hc1k krb5kdc[17020](info): commencing operation
krb5kdc: starting...
W20260926 13:45:12.603420 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:45:12 dist-test-slave-hc1k krb5kdc[17020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:45:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:45:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:45:12.711826 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:40713
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:36749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:40713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:12.811616 17033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:12.811866 17033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:12.811923 17033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:12.815824 17033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:45:12.815869 17033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:12.815893 17033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:12.815924 17033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:45:12.815951 17033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:45:12.820880 17033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:36749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:40713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:40713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:12.821748 17033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:12.822604 17033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:12.829946 17038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:12.829985 17033 server_base.cc:1034] running on GCE node
W20260926 13:45:12.829942 17041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:12.829981 17039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:12.830672 17033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:12.831637 17033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:12.832810 17033 hybrid_clock.cc:648] HybridClock initialized: now 1790430312832796 us; error 26 us; skew 500 ppm
Sep 26 13:45:12 dist-test-slave-hc1k krb5kdc[17020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430312, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:12.838056 17033 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:45:12.838693 17033 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:12.838752 17033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:12.838902 17033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:12.840740 17033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "82dec8df49594aadb4670d5ae0a45824"
format_stamp: "Formatted at 2026-09-26 13:45:12 on dist-test-slave-hc1k"
server_key: "933c35363dbef4bb8d91d9ace6608013"
server_key_iv: "6a5dd7809bcf65a15bb912805065c9ba"
server_key_version: "encryptionkey@0"
I20260926 13:45:12.841203 17033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "82dec8df49594aadb4670d5ae0a45824"
format_stamp: "Formatted at 2026-09-26 13:45:12 on dist-test-slave-hc1k"
server_key: "933c35363dbef4bb8d91d9ace6608013"
server_key_iv: "6a5dd7809bcf65a15bb912805065c9ba"
server_key_version: "encryptionkey@0"
I20260926 13:45:12.844496 17033 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 13:45:12.846567 17047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:12.847683 17033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:12.847784 17033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "82dec8df49594aadb4670d5ae0a45824"
format_stamp: "Formatted at 2026-09-26 13:45:12 on dist-test-slave-hc1k"
server_key: "933c35363dbef4bb8d91d9ace6608013"
server_key_iv: "6a5dd7809bcf65a15bb912805065c9ba"
server_key_version: "encryptionkey@0"
I20260926 13:45:12.847870 17033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:12.864995 17033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:12.867527 17033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:12.867704 17033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:12.874055 17033 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:40713
I20260926 13:45:12.874272 17099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:40713 every 8 connection(s)
I20260926 13:45:12.874626 17033 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:45:12.876214 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 17033
I20260926 13:45:12.876328 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:45:12.876612 12969 external_mini_cluster.cc:1428] Setting key b9161f1c1794de91a7bbf386cc4aaa39
I20260926 13:45:12.877408 17100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:12.883476 17100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82dec8df49594aadb4670d5ae0a45824: Bootstrap starting.
Sep 26 13:45:12 dist-test-slave-hc1k krb5kdc[17020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:12.885568 17100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82dec8df49594aadb4670d5ae0a45824: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:12.886389 17100 log.cc:824] T 00000000000000000000000000000000 P 82dec8df49594aadb4670d5ae0a45824: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:12.888338 17100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82dec8df49594aadb4670d5ae0a45824: No bootstrap required, opened a new log
I20260926 13:45:12.890142 17103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:12.877870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.12.170.126:40713)
0926 13:45:12.878252 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:12.878269 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:12.878307 (+ 38us) server_negotiation.cc:407] Connection header received
0926 13:45:12.879113 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:12.879134 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:12.879528 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:12.879892 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:12.880475 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:12.881868 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:12.882651 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:12.883283 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:12.883442 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:12.885541 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:12.885548 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:12.885564 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:12.885594 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:12.887187 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:12.887555 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:12.887560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:12.887566 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:12.887717 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:12.887942 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:12.887945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:12.887947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:12.888408 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:12.888586 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:12.888813 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 13:45:12.889077 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":122,"threads_started":1}
I20260926 13:45:12.891777 17100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82dec8df49594aadb4670d5ae0a45824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82dec8df49594aadb4670d5ae0a45824" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40713 } }
I20260926 13:45:12.892066 17100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82dec8df49594aadb4670d5ae0a45824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:12.892123 17100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82dec8df49594aadb4670d5ae0a45824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82dec8df49594aadb4670d5ae0a45824, State: Initialized, Role: FOLLOWER
I20260926 13:45:12.892506 17100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82dec8df49594aadb4670d5ae0a45824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82dec8df49594aadb4670d5ae0a45824" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40713 } }
I20260926 13:45:12.892620 17100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82dec8df49594aadb4670d5ae0a45824 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:45:12.892679 17100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82dec8df49594aadb4670d5ae0a45824 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:45:12.892766 17100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82dec8df49594aadb4670d5ae0a45824 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:12.893515 17100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82dec8df49594aadb4670d5ae0a45824 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82dec8df49594aadb4670d5ae0a45824" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40713 } }
I20260926 13:45:12.893824 17100 leader_election.cc:302] T 00000000000000000000000000000000 P 82dec8df49594aadb4670d5ae0a45824 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82dec8df49594aadb4670d5ae0a45824; no voters:
I20260926 13:45:12.894110 17100 leader_election.cc:288] T 00000000000000000000000000000000 P 82dec8df49594aadb4670d5ae0a45824 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:45:12.894197 17105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82dec8df49594aadb4670d5ae0a45824 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:12.894404 17105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82dec8df49594aadb4670d5ae0a45824 [term 1 LEADER]: Becoming Leader. State: Replica: 82dec8df49594aadb4670d5ae0a45824, State: Running, Role: LEADER
I20260926 13:45:12.894707 17105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82dec8df49594aadb4670d5ae0a45824 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82dec8df49594aadb4670d5ae0a45824" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40713 } }
I20260926 13:45:12.895131 17100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82dec8df49594aadb4670d5ae0a45824 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:45:12.896121 17107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82dec8df49594aadb4670d5ae0a45824 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82dec8df49594aadb4670d5ae0a45824. Latest consensus state: current_term: 1 leader_uuid: "82dec8df49594aadb4670d5ae0a45824" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82dec8df49594aadb4670d5ae0a45824" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40713 } } }
I20260926 13:45:12.896232 17107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82dec8df49594aadb4670d5ae0a45824 [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:12.896587 17106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82dec8df49594aadb4670d5ae0a45824 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82dec8df49594aadb4670d5ae0a45824" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82dec8df49594aadb4670d5ae0a45824" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40713 } } }
I20260926 13:45:12.896730 17106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82dec8df49594aadb4670d5ae0a45824 [sys.catalog]: This master's current role is: LEADER
W20260926 13:45:12.899251 17120 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82dec8df49594aadb4670d5ae0a45824: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:45:12.899312 17120 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:45:12.899381 17114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:45:12.900113 17114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:45:12.904737 17114 catalog_manager.cc:1347] Generated new cluster ID: b8be79992df74bbca31e24f5522f4fc7
I20260926 13:45:12.904795 17114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:45:12.910187 17114 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:45:12.911281 17114 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:45:12.916621 17114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82dec8df49594aadb4670d5ae0a45824: Generated new TSK 0
I20260926 13:45:12.917266 17114 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.170.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
I20260926 13:45:12.946655 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:0
--local_ip_for_outbound_sockets=127.12.170.65
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:40713
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:36749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:13.050045 17126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:13.050261 17126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:13.050300 17126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:13.054291 17126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:13.054342 17126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:13.054396 17126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:45:13.059222 17126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:36749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:40713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:13.060087 17126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:13.060978 17126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:13.069288 17132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:13.069283 17131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:13.069284 17134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:13.069370 17126 server_base.cc:1034] running on GCE node
I20260926 13:45:13.069922 17126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:13.070466 17126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:13.071614 17126 hybrid_clock.cc:648] HybridClock initialized: now 1790430313071595 us; error 34 us; skew 500 ppm
Sep 26 13:45:13 dist-test-slave-hc1k krb5kdc[17020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430313, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:13.074064 17126 init.cc:377] Logged in from keytab as kudu/127.12.170.65@KRBTEST.COM (short username kudu)
I20260926 13:45:13.074820 17126 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:13.074901 17126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:13.075126 17126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:13.077015 17126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-0/data/instance:
uuid: "059380dba31244df92621d0a9b15ad8f"
format_stamp: "Formatted at 2026-09-26 13:45:13 on dist-test-slave-hc1k"
server_key: "94a61383433cf4fea06d6cbdc70928c3"
server_key_iv: "5d17d60b6752b605c987b8f55f5fd48d"
server_key_version: "encryptionkey@0"
I20260926 13:45:13.077483 17126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance:
uuid: "059380dba31244df92621d0a9b15ad8f"
format_stamp: "Formatted at 2026-09-26 13:45:13 on dist-test-slave-hc1k"
server_key: "94a61383433cf4fea06d6cbdc70928c3"
server_key_iv: "5d17d60b6752b605c987b8f55f5fd48d"
server_key_version: "encryptionkey@0"
I20260926 13:45:13.080658 17126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 13:45:13.082751 17140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:13.083858 17126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260926 13:45:13.083953 17126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "059380dba31244df92621d0a9b15ad8f"
format_stamp: "Formatted at 2026-09-26 13:45:13 on dist-test-slave-hc1k"
server_key: "94a61383433cf4fea06d6cbdc70928c3"
server_key_iv: "5d17d60b6752b605c987b8f55f5fd48d"
server_key_version: "encryptionkey@0"
I20260926 13:45:13.084036 17126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:13.107045 17126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:13.109822 17126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:13.109972 17126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:13.110464 17126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:13.111177 17126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:13.111219 17126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:13.111258 17126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:13.111287 17126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:13.118852 17126 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:34177
I20260926 13:45:13.118898 17253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:34177 every 8 connection(s)
I20260926 13:45:13.119349 17126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:45:13.121676 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 17126
I20260926 13:45:13.121794 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance
I20260926 13:45:13.122071 12969 external_mini_cluster.cc:1428] Setting key be8c39a96916ded48a474697ed2302e9
Sep 26 13:45:13 dist-test-slave-hc1k krb5kdc[17020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430313, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.170.66@KRBTEST.COM; defaulting to no policy
I20260926 13:45:13.132894 17103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:13.120834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:44349 (local address 127.12.170.126:40713)
0926 13:45:13.120948 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:13.120952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:13.122022 (+ 1070us) server_negotiation.cc:407] Connection header received
0926 13:45:13.122976 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:13.122980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:13.123053 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:13.123159 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:13.124586 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.125401 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.126706 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.127058 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.127160 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:13.129802 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:13.129805 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:13.129807 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:13.129829 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:13.131103 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:13.131688 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:13.131691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:13.131693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:13.131766 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:13.132049 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:13.132051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:13.132052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:13.132204 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:13.132272 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:13.132687 (+ 415us) server_negotiation.cc:299] Negotiation successful
0926 13:45:13.132782 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 13:45:13.133610 17256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:13.121153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40713 (local address 127.12.170.65:44349)
0926 13:45:13.121888 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:13.121940 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:13.122730 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:13.123327 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:13.123335 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:13.123796 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:13.124377 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:13.124397 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.125551 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.125555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:13.126608 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:13.126618 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.127179 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.127182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:13.127308 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:13.127911 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:13.127940 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:13.129649 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:13.131213 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:13.131219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:13.131233 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:13.131545 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:13.131847 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:13.131850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:13.131851 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:13.131980 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:13.132273 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:13.132277 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:13.132492 (+ 215us) client_negotiation.cc:769] Sending connection context
0926 13:45:13.132644 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 13:45:13.132864 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.66@KRBTEST.COM" created.
I20260926 13:45:13.134625 17254 heartbeater.cc:347] Connected to a master server at 127.12.170.126:40713
I20260926 13:45:13.134842 17254 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:13.135285 17254 heartbeater.cc:511] Master 127.12.170.126:40713 requested a full tablet report, sending...
I20260926 13:45:13.136647 17064 ts_manager.cc:194] Registered new tserver with Master: 059380dba31244df92621d0a9b15ad8f (127.12.170.65:34177)
I20260926 13:45:13.137676 17064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.65@KRBTEST.COM'} at 127.12.170.65:44349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
I20260926 13:45:13.146972 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:0
--local_ip_for_outbound_sockets=127.12.170.66
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:40713
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:36749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:13.247267 17259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:13.247478 17259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:13.247517 17259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:13.251354 17259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:13.251400 17259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:13.251457 17259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:45:13.256410 17259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:36749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:40713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:13.257288 17259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:13.258349 17259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:13.266719 17267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:13.266729 17264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:13.266728 17265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:13.266983 17259 server_base.cc:1034] running on GCE node
I20260926 13:45:13.267345 17259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:13.267947 17259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:13.269095 17259 hybrid_clock.cc:648] HybridClock initialized: now 1790430313269074 us; error 33 us; skew 500 ppm
Sep 26 13:45:13 dist-test-slave-hc1k krb5kdc[17020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430313, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:13.271497 17259 init.cc:377] Logged in from keytab as kudu/127.12.170.66@KRBTEST.COM (short username kudu)
I20260926 13:45:13.272158 17259 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:13.272220 17259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:13.272390 17259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:13.274219 17259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-1/data/instance:
uuid: "fb0af04210bc4ff68971ec9bcc8d32b7"
format_stamp: "Formatted at 2026-09-26 13:45:13 on dist-test-slave-hc1k"
server_key: "df46bcec7116df344ae269cf66062cb3"
server_key_iv: "8a0195cb28e0b90497760117b51e85d3"
server_key_version: "encryptionkey@0"
I20260926 13:45:13.274734 17259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance:
uuid: "fb0af04210bc4ff68971ec9bcc8d32b7"
format_stamp: "Formatted at 2026-09-26 13:45:13 on dist-test-slave-hc1k"
server_key: "df46bcec7116df344ae269cf66062cb3"
server_key_iv: "8a0195cb28e0b90497760117b51e85d3"
server_key_version: "encryptionkey@0"
I20260926 13:45:13.278028 17259 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:45:13.280179 17273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:13.281304 17259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:13.281397 17259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "fb0af04210bc4ff68971ec9bcc8d32b7"
format_stamp: "Formatted at 2026-09-26 13:45:13 on dist-test-slave-hc1k"
server_key: "df46bcec7116df344ae269cf66062cb3"
server_key_iv: "8a0195cb28e0b90497760117b51e85d3"
server_key_version: "encryptionkey@0"
I20260926 13:45:13.281488 17259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:13.331475 17259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:13.334367 17259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:13.334551 17259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:13.335125 17259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:13.335958 17259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:13.336005 17259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:13.336058 17259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:13.336082 17259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:13.344386 17259 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:45753
I20260926 13:45:13.344422 17386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:45753 every 8 connection(s)
I20260926 13:45:13.344928 17259 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:45:13.352268 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 17259
I20260926 13:45:13.352376 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance
I20260926 13:45:13.352638 12969 external_mini_cluster.cc:1428] Setting key f56c96c65b3cf51e60c843e54c2c0699
Sep 26 13:45:13 dist-test-slave-hc1k krb5kdc[17020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430313, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:13.357694 17103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:13.346556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:45063 (local address 127.12.170.126:40713)
0926 13:45:13.346674 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:13.346678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:13.347795 (+ 1117us) server_negotiation.cc:407] Connection header received
0926 13:45:13.348640 (+ 845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:13.348643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:13.348701 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:13.348775 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:13.349906 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.350465 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.351688 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.351985 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.352079 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:13.354411 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:13.354414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:13.354417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:13.354440 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:13.355836 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:13.356283 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:13.356287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:13.356289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:13.356376 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:13.356665 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:13.356668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:13.356669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:13.356868 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:13.356961 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:13.357408 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 13:45:13.357536 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 13:45:13.358409 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:13.346869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40713 (local address 127.12.170.66:45063)
0926 13:45:13.347634 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:13.347690 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:13.348443 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:13.348947 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:13.348954 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:13.349290 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:13.349753 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:13.349773 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.350583 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.350585 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:13.351562 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:13.351572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.352145 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.352149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:13.352302 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:13.352859 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:13.352877 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:13.354258 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:13.355944 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:13.355949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:13.355959 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:13.356195 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:13.356458 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:13.356460 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:13.356461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:13.356585 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:13.356974 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:13.356979 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:13.357194 (+ 215us) client_negotiation.cc:769] Sending connection context
0926 13:45:13.357347 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 13:45:13.357592 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":100,"threads_started":1}
I20260926 13:45:13.359602 17387 heartbeater.cc:347] Connected to a master server at 127.12.170.126:40713
I20260926 13:45:13.359820 17387 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:13.360224 17387 heartbeater.cc:511] Master 127.12.170.126:40713 requested a full tablet report, sending...
I20260926 13:45:13.361200 17064 ts_manager.cc:194] Registered new tserver with Master: fb0af04210bc4ff68971ec9bcc8d32b7 (127.12.170.66:45753)
WARNING: no policy specified for kudu/127.12.170.67@KRBTEST.COM; defaulting to no policy
I20260926 13:45:13.361765 17064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.66@KRBTEST.COM'} at 127.12.170.66:45063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
I20260926 13:45:13.376639 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:0
--local_ip_for_outbound_sockets=127.12.170.67
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:40713
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:36749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:13.478668 17392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:13.478883 17392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:13.478950 17392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:13.482901 17392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:13.482956 17392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:13.483033 17392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:45:13.488063 17392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:36749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:40713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:13.488920 17392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:13.489800 17392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:13.498598 17397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:13.498598 17398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:13.498589 17400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:13.498747 17392 server_base.cc:1034] running on GCE node
I20260926 13:45:13.499109 17392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:13.499675 17392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:13.500844 17392 hybrid_clock.cc:648] HybridClock initialized: now 1790430313500825 us; error 35 us; skew 500 ppm
Sep 26 13:45:13 dist-test-slave-hc1k krb5kdc[17020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430313, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:13.503382 17392 init.cc:377] Logged in from keytab as kudu/127.12.170.67@KRBTEST.COM (short username kudu)
I20260926 13:45:13.504046 17392 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:13.504109 17392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:13.504281 17392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:13.506098 17392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-2/data/instance:
uuid: "2612c7156a3042e0ad853ff84792d865"
format_stamp: "Formatted at 2026-09-26 13:45:13 on dist-test-slave-hc1k"
server_key: "3ceca36ec0987cf2590dbd426ae82de6"
server_key_iv: "9b1253a56f5f69cca3e1958b7db8f8b0"
server_key_version: "encryptionkey@0"
I20260926 13:45:13.506598 17392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance:
uuid: "2612c7156a3042e0ad853ff84792d865"
format_stamp: "Formatted at 2026-09-26 13:45:13 on dist-test-slave-hc1k"
server_key: "3ceca36ec0987cf2590dbd426ae82de6"
server_key_iv: "9b1253a56f5f69cca3e1958b7db8f8b0"
server_key_version: "encryptionkey@0"
I20260926 13:45:13.509850 17392 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 13:45:13.511921 17406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:13.513010 17392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 13:45:13.513114 17392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "2612c7156a3042e0ad853ff84792d865"
format_stamp: "Formatted at 2026-09-26 13:45:13 on dist-test-slave-hc1k"
server_key: "3ceca36ec0987cf2590dbd426ae82de6"
server_key_iv: "9b1253a56f5f69cca3e1958b7db8f8b0"
server_key_version: "encryptionkey@0"
I20260926 13:45:13.513192 17392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:13.570593 17392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:13.573582 17392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:13.573755 17392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:13.574321 17392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:13.575102 17392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:13.575146 17392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:13.575197 17392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:13.575227 17392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:13.583149 17392 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:33971
I20260926 13:45:13.583176 17519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:33971 every 8 connection(s)
I20260926 13:45:13.583684 17392 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
Sep 26 13:45:13 dist-test-slave-hc1k krb5kdc[17020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430313, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:13.591969 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 17392
I20260926 13:45:13.592072 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance
I20260926 13:45:13.592317 12969 external_mini_cluster.cc:1428] Setting key 16c68944eab256d87327976840c207cc
I20260926 13:45:13.595568 17103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:13.585246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:55099 (local address 127.12.170.126:40713)
0926 13:45:13.585363 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:13.585367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:13.586514 (+ 1147us) server_negotiation.cc:407] Connection header received
0926 13:45:13.587340 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:13.587342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:13.587387 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:13.587459 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:13.588571 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.589094 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.590100 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.590295 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.590360 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:13.592500 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:13.592503 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:13.592505 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:13.592527 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:13.593972 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:13.594431 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:13.594434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:13.594435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:13.594516 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:13.594746 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:13.594748 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:13.594749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:13.594905 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:13.594968 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:13.595358 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 13:45:13.595457 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:45:13.596467 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:13.585602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40713 (local address 127.12.170.67:55099)
0926 13:45:13.586345 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:13.586414 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:13.587177 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:13.587618 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:13.587624 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:13.587976 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:13.588452 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:13.588465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.589178 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.589180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:13.590017 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:13.590025 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.590400 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.590403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:13.590529 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:13.590999 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:13.591015 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:13.592352 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:13.594086 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:13.594091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:13.594104 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:13.594359 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:13.594579 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:13.594581 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:13.594582 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:13.594693 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:13.594968 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:13.594972 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:13.595167 (+ 195us) client_negotiation.cc:769] Sending connection context
0926 13:45:13.595329 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 13:45:13.595582 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
I20260926 13:45:13.597409 17520 heartbeater.cc:347] Connected to a master server at 127.12.170.126:40713
I20260926 13:45:13.597613 17520 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:13.598002 17520 heartbeater.cc:511] Master 127.12.170.126:40713 requested a full tablet report, sending...
I20260926 13:45:13.598887 17064 ts_manager.cc:194] Registered new tserver with Master: 2612c7156a3042e0ad853ff84792d865 (127.12.170.67:33971)
I20260926 13:45:13.599416 17064 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.67@KRBTEST.COM'} at 127.12.170.67:55099
I20260926 13:45:13.605613 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:45:13.613581 17103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:13.607342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33966 (local address 127.12.170.126:40713)
0926 13:45:13.607452 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:13.607456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:13.607600 (+ 144us) server_negotiation.cc:407] Connection header received
0926 13:45:13.607740 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:13.607742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:13.607778 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:13.607838 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:13.608416 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.608885 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.609689 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.609881 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.609952 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:13.611153 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:13.611155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:13.611157 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:13.611172 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:13.612350 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:13.612695 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:13.612697 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:13.612698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:13.612759 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:13.612964 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:13.612965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:13.612966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:13.613139 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:13.613233 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:13.613400 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 13:45:13.613477 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:45:13.615957 17064 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:45:13.618850 17064 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:45:13.632380 17532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:13.626521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:45753 (local address 127.0.0.1:58694)
0926 13:45:13.627614 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:13.627684 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:13.627806 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:13.628332 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:13.628338 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:13.628358 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:13.628486 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:13.628492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.629807 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.629810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:13.630902 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:13.630926 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.631874 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.631878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:13.631985 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:13.632029 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 13:45:13.632126 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 13:45:13.632242 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":935,"thread_start_us":72,"threads_started":1}
I20260926 13:45:13.632598 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:13.627149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51548 (local address 127.12.170.65:34177)
0926 13:45:13.627763 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:13.627769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:13.627956 (+ 187us) server_negotiation.cc:407] Connection header received
0926 13:45:13.628076 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:13.628081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:13.628269 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:13.628424 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:13.628884 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.629907 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.631332 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.631957 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.632059 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:13.632179 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:13.632286 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 13:45:13.632373 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":105,"threads_started":1}
I20260926 13:45:13.632741 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:13.626658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58694 (local address 127.12.170.66:45753)
0926 13:45:13.627245 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:13.627257 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:13.627823 (+ 566us) server_negotiation.cc:407] Connection header received
0926 13:45:13.627985 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:13.627992 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:13.628190 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:13.628323 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:13.628588 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.629682 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.631032 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.631744 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.631847 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:13.631976 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:13.632516 (+ 540us) server_negotiation.cc:299] Negotiation successful
0926 13:45:13.632606 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":121,"threads_started":1}
I20260926 13:45:13.633020 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:13.626652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:34177 (local address 127.0.0.1:51548)
0926 13:45:13.627855 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:13.627866 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:13.627956 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:13.628571 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:13.628574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:13.628604 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:13.628766 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:13.628772 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.630012 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.630015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:13.631207 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:13.631216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.632074 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.632078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:13.632171 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:13.632193 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 13:45:13.632258 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 13:45:13.632322 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1147,"mutex_wait_us":53,"spinlock_wait_cycles":7424,"thread_start_us":86,"threads_started":1}
I20260926 13:45:13.633898 17531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:13.626864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:33971 (local address 127.0.0.1:54084)
0926 13:45:13.627958 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:13.627971 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:13.628051 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:13.628571 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:13.628574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:13.628598 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:13.628757 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:13.628765 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.631301 (+ 2536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.631304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:13.632364 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:13.632372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.633533 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.633536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:13.633631 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:13.633652 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 13:45:13.633715 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 13:45:13.633777 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"mutex_wait_us":24,"spinlock_wait_cycles":2176,"thread_start_us":77,"threads_started":1}
I20260926 13:45:13.634352 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:13.627266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54084 (local address 127.12.170.67:33971)
0926 13:45:13.627676 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:13.627688 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:13.628050 (+ 362us) server_negotiation.cc:407] Connection header received
0926 13:45:13.628144 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:13.628148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:13.628287 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:13.628418 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:13.630048 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.631176 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.632455 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.633232 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.633327 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:13.633470 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:13.634102 (+ 632us) server_negotiation.cc:299] Negotiation successful
0926 13:45:13.634218 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":161,"threads_started":1}
I20260926 13:45:13.635526 17321 tablet_service.cc:1467] Processing CreateTablet for tablet 1f19e69abbb7419eb6ad9af1a5593434 (DEFAULT_TABLE table=test-table [id=f0bd756d7af04935885896774d3a2f17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:13.635557 17188 tablet_service.cc:1467] Processing CreateTablet for tablet 1f19e69abbb7419eb6ad9af1a5593434 (DEFAULT_TABLE table=test-table [id=f0bd756d7af04935885896774d3a2f17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:13.636437 17321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f19e69abbb7419eb6ad9af1a5593434. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:13.636554 17188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f19e69abbb7419eb6ad9af1a5593434. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:13.636715 17454 tablet_service.cc:1467] Processing CreateTablet for tablet 1f19e69abbb7419eb6ad9af1a5593434 (DEFAULT_TABLE table=test-table [id=f0bd756d7af04935885896774d3a2f17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:13.637605 17454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f19e69abbb7419eb6ad9af1a5593434. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:13.641984 17537 tablet_bootstrap.cc:492] T 1f19e69abbb7419eb6ad9af1a5593434 P fb0af04210bc4ff68971ec9bcc8d32b7: Bootstrap starting.
I20260926 13:45:13.643219 17538 tablet_bootstrap.cc:492] T 1f19e69abbb7419eb6ad9af1a5593434 P 059380dba31244df92621d0a9b15ad8f: Bootstrap starting.
I20260926 13:45:13.643911 17539 tablet_bootstrap.cc:492] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865: Bootstrap starting.
I20260926 13:45:13.644780 17537 tablet_bootstrap.cc:654] T 1f19e69abbb7419eb6ad9af1a5593434 P fb0af04210bc4ff68971ec9bcc8d32b7: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:13.645252 17538 tablet_bootstrap.cc:654] T 1f19e69abbb7419eb6ad9af1a5593434 P 059380dba31244df92621d0a9b15ad8f: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:13.645663 17537 log.cc:824] T 1f19e69abbb7419eb6ad9af1a5593434 P fb0af04210bc4ff68971ec9bcc8d32b7: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:13.645727 17539 tablet_bootstrap.cc:654] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:13.646026 17538 log.cc:824] T 1f19e69abbb7419eb6ad9af1a5593434 P 059380dba31244df92621d0a9b15ad8f: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:13.646690 17539 log.cc:824] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:13.647538 17537 tablet_bootstrap.cc:492] T 1f19e69abbb7419eb6ad9af1a5593434 P fb0af04210bc4ff68971ec9bcc8d32b7: No bootstrap required, opened a new log
I20260926 13:45:13.647729 17537 ts_tablet_manager.cc:1397] T 1f19e69abbb7419eb6ad9af1a5593434 P fb0af04210bc4ff68971ec9bcc8d32b7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 13:45:13.647763 17538 tablet_bootstrap.cc:492] T 1f19e69abbb7419eb6ad9af1a5593434 P 059380dba31244df92621d0a9b15ad8f: No bootstrap required, opened a new log
I20260926 13:45:13.647948 17538 ts_tablet_manager.cc:1397] T 1f19e69abbb7419eb6ad9af1a5593434 P 059380dba31244df92621d0a9b15ad8f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 13:45:13.648527 17539 tablet_bootstrap.cc:492] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865: No bootstrap required, opened a new log
I20260926 13:45:13.648693 17539 ts_tablet_manager.cc:1397] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 13:45:13.650743 17537 raft_consensus.cc:348] T 1f19e69abbb7419eb6ad9af1a5593434 P fb0af04210bc4ff68971ec9bcc8d32b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0af04210bc4ff68971ec9bcc8d32b7" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45753 } } peers { permanent_uuid: "059380dba31244df92621d0a9b15ad8f" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 34177 } } peers { permanent_uuid: "2612c7156a3042e0ad853ff84792d865" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 33971 } }
I20260926 13:45:13.651005 17537 raft_consensus.cc:374] T 1f19e69abbb7419eb6ad9af1a5593434 P fb0af04210bc4ff68971ec9bcc8d32b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:13.651063 17537 raft_consensus.cc:729] T 1f19e69abbb7419eb6ad9af1a5593434 P fb0af04210bc4ff68971ec9bcc8d32b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0af04210bc4ff68971ec9bcc8d32b7, State: Initialized, Role: FOLLOWER
I20260926 13:45:13.651459 17537 consensus_queue.cc:260] T 1f19e69abbb7419eb6ad9af1a5593434 P fb0af04210bc4ff68971ec9bcc8d32b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0af04210bc4ff68971ec9bcc8d32b7" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45753 } } peers { permanent_uuid: "059380dba31244df92621d0a9b15ad8f" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 34177 } } peers { permanent_uuid: "2612c7156a3042e0ad853ff84792d865" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 33971 } }
I20260926 13:45:13.652134 17387 heartbeater.cc:503] Master 127.12.170.126:40713 was elected leader, sending a full tablet report...
I20260926 13:45:13.652137 17538 raft_consensus.cc:348] T 1f19e69abbb7419eb6ad9af1a5593434 P 059380dba31244df92621d0a9b15ad8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0af04210bc4ff68971ec9bcc8d32b7" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45753 } } peers { permanent_uuid: "059380dba31244df92621d0a9b15ad8f" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 34177 } } peers { permanent_uuid: "2612c7156a3042e0ad853ff84792d865" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 33971 } }
I20260926 13:45:13.652364 17538 raft_consensus.cc:374] T 1f19e69abbb7419eb6ad9af1a5593434 P 059380dba31244df92621d0a9b15ad8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:13.652420 17538 raft_consensus.cc:729] T 1f19e69abbb7419eb6ad9af1a5593434 P 059380dba31244df92621d0a9b15ad8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059380dba31244df92621d0a9b15ad8f, State: Initialized, Role: FOLLOWER
I20260926 13:45:13.652431 17537 ts_tablet_manager.cc:1428] T 1f19e69abbb7419eb6ad9af1a5593434 P fb0af04210bc4ff68971ec9bcc8d32b7: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 13:45:13.652812 17538 consensus_queue.cc:260] T 1f19e69abbb7419eb6ad9af1a5593434 P 059380dba31244df92621d0a9b15ad8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0af04210bc4ff68971ec9bcc8d32b7" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45753 } } peers { permanent_uuid: "059380dba31244df92621d0a9b15ad8f" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 34177 } } peers { permanent_uuid: "2612c7156a3042e0ad853ff84792d865" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 33971 } }
I20260926 13:45:13.653436 17254 heartbeater.cc:503] Master 127.12.170.126:40713 was elected leader, sending a full tablet report...
I20260926 13:45:13.653703 17538 ts_tablet_manager.cc:1428] T 1f19e69abbb7419eb6ad9af1a5593434 P 059380dba31244df92621d0a9b15ad8f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 13:45:13.654191 17539 raft_consensus.cc:348] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0af04210bc4ff68971ec9bcc8d32b7" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45753 } } peers { permanent_uuid: "059380dba31244df92621d0a9b15ad8f" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 34177 } } peers { permanent_uuid: "2612c7156a3042e0ad853ff84792d865" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 33971 } }
I20260926 13:45:13.654438 17539 raft_consensus.cc:374] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:13.654501 17539 raft_consensus.cc:729] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2612c7156a3042e0ad853ff84792d865, State: Initialized, Role: FOLLOWER
I20260926 13:45:13.654910 17539 consensus_queue.cc:260] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0af04210bc4ff68971ec9bcc8d32b7" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45753 } } peers { permanent_uuid: "059380dba31244df92621d0a9b15ad8f" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 34177 } } peers { permanent_uuid: "2612c7156a3042e0ad853ff84792d865" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 33971 } }
I20260926 13:45:13.655504 17520 heartbeater.cc:503] Master 127.12.170.126:40713 was elected leader, sending a full tablet report...
I20260926 13:45:13.656246 17539 ts_tablet_manager.cc:1428] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 13:45:13.702085 17545 raft_consensus.cc:482] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:45:13.702256 17545 raft_consensus.cc:504] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0af04210bc4ff68971ec9bcc8d32b7" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45753 } } peers { permanent_uuid: "059380dba31244df92621d0a9b15ad8f" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 34177 } } peers { permanent_uuid: "2612c7156a3042e0ad853ff84792d865" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 33971 } }
I20260926 13:45:13.703370 17545 leader_election.cc:288] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb0af04210bc4ff68971ec9bcc8d32b7 (127.12.170.66:45753), 059380dba31244df92621d0a9b15ad8f (127.12.170.65:34177)
I20260926 13:45:13.707782 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:13.703818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:50759 (local address 127.12.170.66:45753)
0926 13:45:13.703934 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:13.703937 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:13.703948 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:45:13.704011 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:13.704013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:13.704049 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:13.704121 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:13.704356 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.705098 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.706555 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.707179 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.707286 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:13.707335 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:13.707580 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 13:45:13.707642 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 13:45:13.707782 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:13.703758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:46887 (local address 127.12.170.65:34177)
0926 13:45:13.703865 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:13.703870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:13.704094 (+ 224us) server_negotiation.cc:407] Connection header received
0926 13:45:13.704176 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:13.704179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:13.704221 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:13.704274 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:13.704498 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.705204 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.706555 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.707179 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.707286 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:13.707335 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:13.707581 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 13:45:13.707643 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 13:45:13.707813 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:13.703591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:45753 (local address 127.12.170.67:50759)
0926 13:45:13.703793 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:13.703808 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:13.703924 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:13.704139 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:13.704142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:13.704156 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:13.704258 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:13.704264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.705197 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.705199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:13.706447 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:13.706458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.707281 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.707285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:13.707465 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:13.707500 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 13:45:13.707561 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 13:45:13.707667 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":76}
I20260926 13:45:13.707820 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:13.703627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:34177 (local address 127.12.170.67:46887)
0926 13:45:13.704016 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:13.704027 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:13.704103 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:13.704294 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:13.704297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:13.704311 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:13.704416 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:13.704421 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.705282 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.705285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:13.706463 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:13.706470 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.707281 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.707285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:13.707439 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:13.707486 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 13:45:13.707561 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 13:45:13.707665 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":26,"thread_start_us":114,"threads_started":1}
I20260926 13:45:13.708441 17341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f19e69abbb7419eb6ad9af1a5593434" candidate_uuid: "2612c7156a3042e0ad853ff84792d865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb0af04210bc4ff68971ec9bcc8d32b7" is_pre_election: true
I20260926 13:45:13.708612 17208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f19e69abbb7419eb6ad9af1a5593434" candidate_uuid: "2612c7156a3042e0ad853ff84792d865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059380dba31244df92621d0a9b15ad8f" is_pre_election: true
I20260926 13:45:13.708741 17341 raft_consensus.cc:2454] T 1f19e69abbb7419eb6ad9af1a5593434 P fb0af04210bc4ff68971ec9bcc8d32b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2612c7156a3042e0ad853ff84792d865 in term 0.
I20260926 13:45:13.708842 17208 raft_consensus.cc:2454] T 1f19e69abbb7419eb6ad9af1a5593434 P 059380dba31244df92621d0a9b15ad8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2612c7156a3042e0ad853ff84792d865 in term 0.
I20260926 13:45:13.709194 17408 leader_election.cc:302] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2612c7156a3042e0ad853ff84792d865, fb0af04210bc4ff68971ec9bcc8d32b7; no voters:
I20260926 13:45:13.709444 17545 raft_consensus.cc:2781] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:45:13.709517 17545 raft_consensus.cc:482] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:45:13.709569 17545 raft_consensus.cc:3037] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:13.710490 17545 raft_consensus.cc:504] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0af04210bc4ff68971ec9bcc8d32b7" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45753 } } peers { permanent_uuid: "059380dba31244df92621d0a9b15ad8f" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 34177 } } peers { permanent_uuid: "2612c7156a3042e0ad853ff84792d865" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 33971 } }
I20260926 13:45:13.710784 17545 leader_election.cc:288] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865 [CANDIDATE]: Term 1 election: Requested vote from peers fb0af04210bc4ff68971ec9bcc8d32b7 (127.12.170.66:45753), 059380dba31244df92621d0a9b15ad8f (127.12.170.65:34177)
I20260926 13:45:13.711066 17208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f19e69abbb7419eb6ad9af1a5593434" candidate_uuid: "2612c7156a3042e0ad853ff84792d865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059380dba31244df92621d0a9b15ad8f"
I20260926 13:45:13.711102 17341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f19e69abbb7419eb6ad9af1a5593434" candidate_uuid: "2612c7156a3042e0ad853ff84792d865" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb0af04210bc4ff68971ec9bcc8d32b7"
I20260926 13:45:13.711190 17208 raft_consensus.cc:3037] T 1f19e69abbb7419eb6ad9af1a5593434 P 059380dba31244df92621d0a9b15ad8f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:13.711199 17341 raft_consensus.cc:3037] T 1f19e69abbb7419eb6ad9af1a5593434 P fb0af04210bc4ff68971ec9bcc8d32b7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:13.712395 17208 raft_consensus.cc:2454] T 1f19e69abbb7419eb6ad9af1a5593434 P 059380dba31244df92621d0a9b15ad8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2612c7156a3042e0ad853ff84792d865 in term 1.
I20260926 13:45:13.712399 17341 raft_consensus.cc:2454] T 1f19e69abbb7419eb6ad9af1a5593434 P fb0af04210bc4ff68971ec9bcc8d32b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2612c7156a3042e0ad853ff84792d865 in term 1.
I20260926 13:45:13.712679 17410 leader_election.cc:302] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 059380dba31244df92621d0a9b15ad8f, 2612c7156a3042e0ad853ff84792d865, fb0af04210bc4ff68971ec9bcc8d32b7; no voters:
I20260926 13:45:13.712834 17545 raft_consensus.cc:2781] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:13.713042 17545 raft_consensus.cc:686] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865 [term 1 LEADER]: Becoming Leader. State: Replica: 2612c7156a3042e0ad853ff84792d865, State: Running, Role: LEADER
I20260926 13:45:13.713322 17545 consensus_queue.cc:237] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0af04210bc4ff68971ec9bcc8d32b7" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45753 } } peers { permanent_uuid: "059380dba31244df92621d0a9b15ad8f" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 34177 } } peers { permanent_uuid: "2612c7156a3042e0ad853ff84792d865" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 33971 } }
I20260926 13:45:13.716223 17063 catalog_manager.cc:5495] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2612c7156a3042e0ad853ff84792d865 (127.12.170.67). New cstate: current_term: 1 leader_uuid: "2612c7156a3042e0ad853ff84792d865" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb0af04210bc4ff68971ec9bcc8d32b7" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "059380dba31244df92621d0a9b15ad8f" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 34177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2612c7156a3042e0ad853ff84792d865" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 33971 } health_report { overall_health: HEALTHY } } }
I20260926 13:45:13.728716 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:13.725808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54096 (local address 127.12.170.67:33971)
0926 13:45:13.725896 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:13.725900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:13.725911 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:45:13.726038 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:13.726040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:13.726083 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:13.726155 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:45:13.726328 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.726789 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.727553 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:13.727791 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:13.727859 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:13.728092 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:45:13.728439 (+ 347us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:45:13.728517 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:13.728544 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 13:45:13.728585 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 13:45:13.733589 17341 raft_consensus.cc:1261] T 1f19e69abbb7419eb6ad9af1a5593434 P fb0af04210bc4ff68971ec9bcc8d32b7 [term 1 FOLLOWER]: Refusing update from remote peer 2612c7156a3042e0ad853ff84792d865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:13.733589 17208 raft_consensus.cc:1261] T 1f19e69abbb7419eb6ad9af1a5593434 P 059380dba31244df92621d0a9b15ad8f [term 1 FOLLOWER]: Refusing update from remote peer 2612c7156a3042e0ad853ff84792d865: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:13.734179 17545 consensus_queue.cc:1035] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "059380dba31244df92621d0a9b15ad8f" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 34177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:13.734335 17547 consensus_queue.cc:1035] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0af04210bc4ff68971ec9bcc8d32b7" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:13.741871 17551 mvcc.cc:205] Tried to move back new op lower bound from 7333602565046890496 to 7333602564972019712. Current Snapshot: MvccSnapshot[applied={T|T < 7333602565046890496}]
I20260926 13:45:13.748603 17063 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:45:13.748720 17063 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:45:13.750758 17063 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82dec8df49594aadb4670d5ae0a45824: Sending DeleteTablet for 3 replicas of tablet 1f19e69abbb7419eb6ad9af1a5593434
I20260926 13:45:13.751397 17321 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f19e69abbb7419eb6ad9af1a5593434 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:13 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:58694
I20260926 13:45:13.751634 17454 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f19e69abbb7419eb6ad9af1a5593434 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:13 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:54084
I20260926 13:45:13.751802 17188 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f19e69abbb7419eb6ad9af1a5593434 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:13 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:51548
I20260926 13:45:13.752213 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 17126
I20260926 13:45:13.752219 17561 tablet_replica.cc:330] T 1f19e69abbb7419eb6ad9af1a5593434 P fb0af04210bc4ff68971ec9bcc8d32b7: stopping tablet replica
I20260926 13:45:13.752604 17561 raft_consensus.cc:2229] T 1f19e69abbb7419eb6ad9af1a5593434 P fb0af04210bc4ff68971ec9bcc8d32b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:45:13.752784 17559 tablet_replica.cc:330] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865: stopping tablet replica
I20260926 13:45:13.752986 17561 raft_consensus.cc:2258] T 1f19e69abbb7419eb6ad9af1a5593434 P fb0af04210bc4ff68971ec9bcc8d32b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:13.753037 17559 raft_consensus.cc:2229] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:45:13.753350 17559 raft_consensus.cc:2258] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:13.754614 17559 ts_tablet_manager.cc:1905] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:13.755873 17561 ts_tablet_manager.cc:1905] T 1f19e69abbb7419eb6ad9af1a5593434 P fb0af04210bc4ff68971ec9bcc8d32b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:13.757097 17559 ts_tablet_manager.cc:1918] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:45:13.757182 17559 log.cc:1196] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/1f19e69abbb7419eb6ad9af1a5593434
I20260926 13:45:13.757462 17559 ts_tablet_manager.cc:1939] T 1f19e69abbb7419eb6ad9af1a5593434 P 2612c7156a3042e0ad853ff84792d865: Deleting consensus metadata
I20260926 13:45:13.758481 17561 ts_tablet_manager.cc:1918] T 1f19e69abbb7419eb6ad9af1a5593434 P fb0af04210bc4ff68971ec9bcc8d32b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:45:13.758572 17050 catalog_manager.cc:4841] TS 2612c7156a3042e0ad853ff84792d865 (127.12.170.67:33971): tablet 1f19e69abbb7419eb6ad9af1a5593434 (table test-table [id=f0bd756d7af04935885896774d3a2f17]) successfully deleted
I20260926 13:45:13.758618 17561 log.cc:1196] T 1f19e69abbb7419eb6ad9af1a5593434 P fb0af04210bc4ff68971ec9bcc8d32b7: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790430291939677-12969-0/minicluster-data/ts-1/wal/wals/1f19e69abbb7419eb6ad9af1a5593434
I20260926 13:45:13.758982 17561 ts_tablet_manager.cc:1939] T 1f19e69abbb7419eb6ad9af1a5593434 P fb0af04210bc4ff68971ec9bcc8d32b7: Deleting consensus metadata
I20260926 13:45:13.759986 17049 catalog_manager.cc:4841] TS fb0af04210bc4ff68971ec9bcc8d32b7 (127.12.170.66:45753): tablet 1f19e69abbb7419eb6ad9af1a5593434 (table test-table [id=f0bd756d7af04935885896774d3a2f17]) successfully deleted
W20260926 13:45:13.762247 17051 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.170.65:34177 (error 108)
I20260926 13:45:13.762921 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 17259
I20260926 13:45:13.763129 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:13.762859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:34177 (local address 127.0.0.1:51552)
0926 13:45:13.762991 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:13.763054 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.65:34177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 13:45:13.764391 17051 catalog_manager.cc:4584] TS 059380dba31244df92621d0a9b15ad8f (127.12.170.65:34177): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1f19e69abbb7419eb6ad9af1a5593434: Network error: Client connection negotiation failed: client connection to 127.12.170.65:34177: connect: Connection refused (error 111)
I20260926 13:45:13.767638 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 17392
I20260926 13:45:13.771747 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 17033
2026-09-26T13:45:13Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2110 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T13:45:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:45:13Z Disabled control of system clock
I20260926 13:45:13.798012 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:44583
--webserver_interface=127.12.170.126
--webserver_port=0
--builtin_ntp_servers=127.12.170.84:33591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:44583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 13:45:13.897704 17566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:13.897923 17566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:13.897955 17566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:13.901891 17566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:45:13.901947 17566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:13.901961 17566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:13.901975 17566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:45:13.901988 17566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:45:13.906638 17566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:33591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:44583
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:44583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:13.907586 17566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:13.908869 17566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:13.916646 17571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:13.916823 17566 server_base.cc:1034] running on GCE node
W20260926 13:45:13.916658 17574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:13.916658 17572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:13.917447 17566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:13.918459 17566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:13.919624 17566 hybrid_clock.cc:648] HybridClock initialized: now 1790430313919609 us; error 25 us; skew 500 ppm
I20260926 13:45:13.921465 17566 webserver.cc:460] Webserver started at http://127.12.170.126:44017/ using document root <none> and password file <none>
I20260926 13:45:13.922031 17566 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:13.922101 17566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:13.922299 17566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:13.924319 17566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "84e1ef3a4dfe4cc88086c7aed189b64c"
format_stamp: "Formatted at 2026-09-26 13:45:13 on dist-test-slave-hc1k"
server_key: "6ce99197dfd8539e166d3b92a2be7455"
server_key_iv: "8bb0040d7d34bd6923f639bf42c2a65c"
server_key_version: "encryptionkey@0"
I20260926 13:45:13.924868 17566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "84e1ef3a4dfe4cc88086c7aed189b64c"
format_stamp: "Formatted at 2026-09-26 13:45:13 on dist-test-slave-hc1k"
server_key: "6ce99197dfd8539e166d3b92a2be7455"
server_key_iv: "8bb0040d7d34bd6923f639bf42c2a65c"
server_key_version: "encryptionkey@0"
I20260926 13:45:13.928406 17566 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:45:13.930567 17580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:13.931675 17566 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260926 13:45:13.931800 17566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "84e1ef3a4dfe4cc88086c7aed189b64c"
format_stamp: "Formatted at 2026-09-26 13:45:13 on dist-test-slave-hc1k"
server_key: "6ce99197dfd8539e166d3b92a2be7455"
server_key_iv: "8bb0040d7d34bd6923f639bf42c2a65c"
server_key_version: "encryptionkey@0"
I20260926 13:45:13.931895 17566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:13.936162 17566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:13.936719 17566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:13.936878 17566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:13.943218 17566 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:44583
I20260926 13:45:13.943229 17632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:44583 every 8 connection(s)
I20260926 13:45:13.944082 17566 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:45:13.946930 17633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:13.951939 17633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84e1ef3a4dfe4cc88086c7aed189b64c: Bootstrap starting.
I20260926 13:45:13.952359 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 17566
I20260926 13:45:13.952463 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:45:13.952713 12969 external_mini_cluster.cc:1428] Setting key 46c3bbbdf5f279b43c4711b888945e7f
I20260926 13:45:13.954452 17633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84e1ef3a4dfe4cc88086c7aed189b64c: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:13.955112 17633 log.cc:824] T 00000000000000000000000000000000 P 84e1ef3a4dfe4cc88086c7aed189b64c: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:13.957011 17633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84e1ef3a4dfe4cc88086c7aed189b64c: No bootstrap required, opened a new log
I20260926 13:45:13.958092 17636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:13.953936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50992 (local address 127.12.170.126:44583)
0926 13:45:13.954604 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:13.954620 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:13.954655 (+ 35us) server_negotiation.cc:407] Connection header received
0926 13:45:13.955313 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:13.955342 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:13.955720 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:13.956081 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:13.956882 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:13.956886 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:13.956898 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:45:13.956909 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:13.956968 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:45:13.956985 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:13.957088 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:13.957220 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 13:45:13.957450 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":142,"threads_started":1}
I20260926 13:45:13.960675 17633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84e1ef3a4dfe4cc88086c7aed189b64c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84e1ef3a4dfe4cc88086c7aed189b64c" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 44583 } }
I20260926 13:45:13.960866 17633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84e1ef3a4dfe4cc88086c7aed189b64c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:13.960938 17633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84e1ef3a4dfe4cc88086c7aed189b64c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84e1ef3a4dfe4cc88086c7aed189b64c, State: Initialized, Role: FOLLOWER
I20260926 13:45:13.961301 17633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84e1ef3a4dfe4cc88086c7aed189b64c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84e1ef3a4dfe4cc88086c7aed189b64c" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 44583 } }
I20260926 13:45:13.961419 17633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84e1ef3a4dfe4cc88086c7aed189b64c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:45:13.961479 17633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84e1ef3a4dfe4cc88086c7aed189b64c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:45:13.961572 17633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84e1ef3a4dfe4cc88086c7aed189b64c [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:13.962347 17633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84e1ef3a4dfe4cc88086c7aed189b64c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84e1ef3a4dfe4cc88086c7aed189b64c" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 44583 } }
I20260926 13:45:13.962649 17633 leader_election.cc:302] T 00000000000000000000000000000000 P 84e1ef3a4dfe4cc88086c7aed189b64c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84e1ef3a4dfe4cc88086c7aed189b64c; no voters:
I20260926 13:45:13.962951 17633 leader_election.cc:288] T 00000000000000000000000000000000 P 84e1ef3a4dfe4cc88086c7aed189b64c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:45:13.963035 17638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84e1ef3a4dfe4cc88086c7aed189b64c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:13.963248 17638 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84e1ef3a4dfe4cc88086c7aed189b64c [term 1 LEADER]: Becoming Leader. State: Replica: 84e1ef3a4dfe4cc88086c7aed189b64c, State: Running, Role: LEADER
I20260926 13:45:13.963564 17638 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84e1ef3a4dfe4cc88086c7aed189b64c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84e1ef3a4dfe4cc88086c7aed189b64c" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 44583 } }
I20260926 13:45:13.964102 17633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84e1ef3a4dfe4cc88086c7aed189b64c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:45:13.964962 17640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84e1ef3a4dfe4cc88086c7aed189b64c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84e1ef3a4dfe4cc88086c7aed189b64c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84e1ef3a4dfe4cc88086c7aed189b64c" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 44583 } } }
I20260926 13:45:13.965183 17639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84e1ef3a4dfe4cc88086c7aed189b64c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84e1ef3a4dfe4cc88086c7aed189b64c. Latest consensus state: current_term: 1 leader_uuid: "84e1ef3a4dfe4cc88086c7aed189b64c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84e1ef3a4dfe4cc88086c7aed189b64c" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 44583 } } }
I20260926 13:45:13.965263 17640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84e1ef3a4dfe4cc88086c7aed189b64c [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:13.965312 17639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84e1ef3a4dfe4cc88086c7aed189b64c [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:13.965659 17647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:45:13.968122 17647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:45:13.972810 17647 catalog_manager.cc:1347] Generated new cluster ID: 14fc1436604a458a8ede9baad343ca4c
I20260926 13:45:13.972875 17647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:45:13.994267 17647 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:45:13.994561 17647 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:45:14.001698 17647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84e1ef3a4dfe4cc88086c7aed189b64c: Generated new TSK 0
I20260926 13:45:14.002420 17647 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:45:14.012596 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:0
--local_ip_for_outbound_sockets=127.12.170.65
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:44583
--builtin_ntp_servers=127.12.170.84:33591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 13:45:14.115453 17657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:14.115698 17657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:14.115760 17657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:14.119693 17657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:14.119735 17657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:14.119800 17657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:45:14.124285 17657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:33591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:44583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:14.125144 17657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:14.126327 17657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:14.135618 17665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:14.135612 17662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:14.135828 17657 server_base.cc:1034] running on GCE node
W20260926 13:45:14.135751 17663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:14.136300 17657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:14.136834 17657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:14.137998 17657 hybrid_clock.cc:648] HybridClock initialized: now 1790430314137979 us; error 30 us; skew 500 ppm
I20260926 13:45:14.139524 17657 webserver.cc:460] Webserver started at http://127.12.170.65:42165/ using document root <none> and password file <none>
I20260926 13:45:14.140076 17657 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:14.140134 17657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:14.140311 17657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:14.142148 17657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-0/data/instance:
uuid: "1c89af6d577f482181255aceae354972"
format_stamp: "Formatted at 2026-09-26 13:45:14 on dist-test-slave-hc1k"
server_key: "fdf805d1d69c848950574bd53ff300d1"
server_key_iv: "789d3d11c89d446363525ce389d16190"
server_key_version: "encryptionkey@0"
I20260926 13:45:14.142640 17657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance:
uuid: "1c89af6d577f482181255aceae354972"
format_stamp: "Formatted at 2026-09-26 13:45:14 on dist-test-slave-hc1k"
server_key: "fdf805d1d69c848950574bd53ff300d1"
server_key_iv: "789d3d11c89d446363525ce389d16190"
server_key_version: "encryptionkey@0"
I20260926 13:45:14.145854 17657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:45:14.148098 17671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:14.149173 17657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:45:14.149269 17657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "1c89af6d577f482181255aceae354972"
format_stamp: "Formatted at 2026-09-26 13:45:14 on dist-test-slave-hc1k"
server_key: "fdf805d1d69c848950574bd53ff300d1"
server_key_iv: "789d3d11c89d446363525ce389d16190"
server_key_version: "encryptionkey@0"
I20260926 13:45:14.149356 17657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:14.153473 17657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:14.153987 17657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:14.154124 17657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:14.154577 17657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:14.155354 17657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:14.155400 17657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:14.155445 17657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:14.155470 17657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:14.163210 17657 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:38935
I20260926 13:45:14.163239 17784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:38935 every 8 connection(s)
I20260926 13:45:14.164007 17657 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:45:14.167641 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 17657
I20260926 13:45:14.167753 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance
I20260926 13:45:14.167999 12969 external_mini_cluster.cc:1428] Setting key d7d22ffbfcb6aea37a7d61ff15d92afb
I20260926 13:45:14.169574 17636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:14.165587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:56841 (local address 127.12.170.126:44583)
0926 13:45:14.165712 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:14.165716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:14.166741 (+ 1025us) server_negotiation.cc:407] Connection header received
0926 13:45:14.167987 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:14.167990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:14.168033 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:14.168110 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:14.169141 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:14.169144 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:14.169147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:45:14.169158 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:14.169178 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:45:14.169186 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:14.169263 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:14.169413 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 13:45:14.169454 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 13:45:14.169814 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:0
--local_ip_for_outbound_sockets=127.12.170.66
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:44583
--builtin_ntp_servers=127.12.170.84:33591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 13:45:14.170332 17787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:14.165991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:44583 (local address 127.12.170.65:56841)
0926 13:45:14.166621 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:14.166658 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:14.167787 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:14.168298 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:14.168306 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:14.168404 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:14.168887 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:45:14.168888 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:14.168940 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:45:14.169004 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:14.169269 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:14.169277 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:14.169283 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 13:45:14.169421 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 13:45:14.169590 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":110,"threads_started":1}
I20260926 13:45:14.171446 17785 heartbeater.cc:347] Connected to a master server at 127.12.170.126:44583
I20260926 13:45:14.171636 17785 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:14.172000 17785 heartbeater.cc:511] Master 127.12.170.126:44583 requested a full tablet report, sending...
I20260926 13:45:14.173285 17597 ts_manager.cc:194] Registered new tserver with Master: 1c89af6d577f482181255aceae354972 (127.12.170.65:38935)
W20260926 13:45:14.275489 17788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:14.275722 17788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:14.275772 17788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:14.279564 17788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:14.279606 17788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:14.279697 17788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:45:14.284168 17788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:33591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:44583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:14.285013 17788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:14.286170 17788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:14.294647 17793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:14.294692 17794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:14.294747 17788 server_base.cc:1034] running on GCE node
W20260926 13:45:14.294697 17796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:14.295167 17788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:14.295725 17788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:14.296913 17788 hybrid_clock.cc:648] HybridClock initialized: now 1790430314296887 us; error 37 us; skew 500 ppm
I20260926 13:45:14.298475 17788 webserver.cc:460] Webserver started at http://127.12.170.66:45765/ using document root <none> and password file <none>
I20260926 13:45:14.298965 17788 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:14.299026 17788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:14.299190 17788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:14.301067 17788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-1/data/instance:
uuid: "ccad323169d34a36b8896cd0e3e0db48"
format_stamp: "Formatted at 2026-09-26 13:45:14 on dist-test-slave-hc1k"
server_key: "3a34a1fa597b27ee6243518c6e720784"
server_key_iv: "f722f3523feb88afd930ed48caa94f1e"
server_key_version: "encryptionkey@0"
I20260926 13:45:14.301543 17788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance:
uuid: "ccad323169d34a36b8896cd0e3e0db48"
format_stamp: "Formatted at 2026-09-26 13:45:14 on dist-test-slave-hc1k"
server_key: "3a34a1fa597b27ee6243518c6e720784"
server_key_iv: "f722f3523feb88afd930ed48caa94f1e"
server_key_version: "encryptionkey@0"
I20260926 13:45:14.304831 17788 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 13:45:14.307036 17802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:14.308172 17788 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 13:45:14.308270 17788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "ccad323169d34a36b8896cd0e3e0db48"
format_stamp: "Formatted at 2026-09-26 13:45:14 on dist-test-slave-hc1k"
server_key: "3a34a1fa597b27ee6243518c6e720784"
server_key_iv: "f722f3523feb88afd930ed48caa94f1e"
server_key_version: "encryptionkey@0"
I20260926 13:45:14.308355 17788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:14.312379 17788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:14.312885 17788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:14.313023 17788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:14.313483 17788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:14.314271 17788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:14.314313 17788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:14.314347 17788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:14.314366 17788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:14.321801 17788 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:35773
I20260926 13:45:14.321834 17915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:35773 every 8 connection(s)
I20260926 13:45:14.322552 17788 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:45:14.326488 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 17788
I20260926 13:45:14.326586 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance
I20260926 13:45:14.326809 12969 external_mini_cluster.cc:1428] Setting key 101e8bd073510dc448697ba644582dae
I20260926 13:45:14.327806 17636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:14.324071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:48061 (local address 127.12.170.126:44583)
0926 13:45:14.324186 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:14.324190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:14.325114 (+ 924us) server_negotiation.cc:407] Connection header received
0926 13:45:14.326266 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:14.326269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:14.326312 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:14.326384 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:14.327375 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:14.327379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:14.327382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:45:14.327393 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:14.327422 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:45:14.327431 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:14.327504 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:14.327621 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 13:45:14.327682 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 13:45:14.328549 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:0
--local_ip_for_outbound_sockets=127.12.170.67
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:44583
--builtin_ntp_servers=127.12.170.84:33591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 13:45:14.328612 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:14.324408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:44583 (local address 127.12.170.66:48061)
0926 13:45:14.324986 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:14.325024 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:14.326081 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:14.326523 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:14.326531 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:14.326634 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:14.327142 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:45:14.327144 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:14.327180 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:45:14.327242 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:14.327499 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:14.327507 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:14.327512 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:45:14.327675 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 13:45:14.327988 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":99,"threads_started":1}
I20260926 13:45:14.329536 17916 heartbeater.cc:347] Connected to a master server at 127.12.170.126:44583
I20260926 13:45:14.329727 17916 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:14.330055 17916 heartbeater.cc:511] Master 127.12.170.126:44583 requested a full tablet report, sending...
I20260926 13:45:14.330915 17597 ts_manager.cc:194] Registered new tserver with Master: ccad323169d34a36b8896cd0e3e0db48 (127.12.170.66:35773)
W20260926 13:45:14.429661 17919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:14.429888 17919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:14.429940 17919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:14.433840 17919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:14.433887 17919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:14.433957 17919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:45:14.438477 17919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:33591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:44583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:14.439368 17919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:14.440608 17919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:14.449133 17924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:14.449182 17925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:14.449241 17919 server_base.cc:1034] running on GCE node
W20260926 13:45:14.449191 17927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:14.449746 17919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:14.450326 17919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:14.451519 17919 hybrid_clock.cc:648] HybridClock initialized: now 1790430314451505 us; error 28 us; skew 500 ppm
I20260926 13:45:14.453226 17919 webserver.cc:460] Webserver started at http://127.12.170.67:36763/ using document root <none> and password file <none>
I20260926 13:45:14.453715 17919 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:14.453775 17919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:14.453941 17919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:14.455885 17919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-2/data/instance:
uuid: "556c21109de24aa9b1aa842df644bc1e"
format_stamp: "Formatted at 2026-09-26 13:45:14 on dist-test-slave-hc1k"
server_key: "d8d8255073e44104a356de6ed9cebaee"
server_key_iv: "633eaacc92dc07bbc4e763091800a26a"
server_key_version: "encryptionkey@0"
I20260926 13:45:14.456429 17919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance:
uuid: "556c21109de24aa9b1aa842df644bc1e"
format_stamp: "Formatted at 2026-09-26 13:45:14 on dist-test-slave-hc1k"
server_key: "d8d8255073e44104a356de6ed9cebaee"
server_key_iv: "633eaacc92dc07bbc4e763091800a26a"
server_key_version: "encryptionkey@0"
I20260926 13:45:14.459854 17919 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:45:14.462057 17933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:14.463151 17919 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:14.463241 17919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "556c21109de24aa9b1aa842df644bc1e"
format_stamp: "Formatted at 2026-09-26 13:45:14 on dist-test-slave-hc1k"
server_key: "d8d8255073e44104a356de6ed9cebaee"
server_key_iv: "633eaacc92dc07bbc4e763091800a26a"
server_key_version: "encryptionkey@0"
I20260926 13:45:14.463327 17919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:14.467760 17919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:14.468266 17919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:14.468413 17919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:14.468866 17919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:14.469673 17919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:14.469715 17919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:14.469761 17919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:14.469789 17919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:14.477301 17919 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:39165
I20260926 13:45:14.477329 18046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:39165 every 8 connection(s)
I20260926 13:45:14.478019 17919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:45:14.483006 17636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:14.479502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:34811 (local address 127.12.170.126:44583)
0926 13:45:14.479623 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:14.479627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:14.480576 (+ 949us) server_negotiation.cc:407] Connection header received
0926 13:45:14.481712 (+ 1136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:14.481714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:14.481749 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:14.481814 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:14.482660 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:14.482663 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:14.482665 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:45:14.482675 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:14.482694 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:45:14.482702 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:14.482768 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:14.482875 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 13:45:14.482908 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:45:14.483577 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 17919
I20260926 13:45:14.483714 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance
I20260926 13:45:14.483897 18049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:14.479872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:44583 (local address 127.12.170.67:34811)
0926 13:45:14.480455 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:14.480489 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:14.481537 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:14.481939 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:14.481945 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:14.482022 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:14.482443 (+ 421us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:45:14.482445 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:14.482478 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:45:14.482539 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:14.482778 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:14.482782 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:14.482786 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:45:14.482902 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 13:45:14.483072 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":100,"threads_started":1}
I20260926 13:45:14.484020 12969 external_mini_cluster.cc:1428] Setting key f2f20f7a59ce6b2e897cf444f3e490c4
I20260926 13:45:14.484921 18047 heartbeater.cc:347] Connected to a master server at 127.12.170.126:44583
I20260926 13:45:14.485126 18047 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:14.485517 18047 heartbeater.cc:511] Master 127.12.170.126:44583 requested a full tablet report, sending...
I20260926 13:45:14.486366 17597 ts_manager.cc:194] Registered new tserver with Master: 556c21109de24aa9b1aa842df644bc1e (127.12.170.67:39165)
I20260926 13:45:14.497164 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:45:14.500248 17636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:14.498822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50998 (local address 127.12.170.126:44583)
0926 13:45:14.498944 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:14.498948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:14.499097 (+ 149us) server_negotiation.cc:407] Connection header received
0926 13:45:14.499199 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:14.499202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:14.499237 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:14.499295 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:14.499917 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:14.499920 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:14.499923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:45:14.499934 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:14.499953 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:45:14.499960 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:14.500025 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:14.500098 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 13:45:14.500136 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:45:14.502607 17597 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50998:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:45:14.505445 17597 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:45:14.515630 18059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:14.512658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:38935 (local address 127.0.0.1:34352)
0926 13:45:14.513405 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:14.513417 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:14.513482 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:14.514135 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:14.514138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:14.514178 (+ 40us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:14.514615 (+ 437us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:45:14.514617 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:14.514634 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:45:14.514697 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:14.515292 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:14.515301 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:14.515306 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:45:14.515413 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 13:45:14.515473 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":71,"threads_started":1}
I20260926 13:45:14.515801 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:14.512647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60172 (local address 127.12.170.67:39165)
0926 13:45:14.513684 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:14.513695 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:14.513717 (+ 22us) server_negotiation.cc:407] Connection header received
0926 13:45:14.513760 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:14.513765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:14.513948 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:14.514411 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:14.515177 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:14.515182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:14.515193 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:45:14.515203 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:14.515244 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:45:14.515266 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:14.515549 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:14.515592 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 13:45:14.515675 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":949,"thread_start_us":118,"threads_started":1}
I20260926 13:45:14.515642 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:14.512523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:39165 (local address 127.0.0.1:60172)
0926 13:45:14.513000 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:14.513034 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:14.513102 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:14.514103 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:14.514109 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:14.514178 (+ 69us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:14.514761 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:45:14.514763 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:14.514774 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:45:14.514801 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:14.515360 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:14.515363 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:14.515366 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:45:14.515412 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 13:45:14.515473 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":62,"threads_started":1}
I20260926 13:45:14.516116 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:14.513165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:35773 (local address 127.0.0.1:46370)
0926 13:45:14.513417 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:14.513430 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:14.513502 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:14.515054 (+ 1552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:14.515057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:14.515071 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:14.515457 (+ 386us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:45:14.515459 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:14.515468 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:45:14.515492 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:14.515809 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:14.515813 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:14.515815 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:45:14.515896 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 13:45:14.516017 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":77,"threads_started":1}
I20260926 13:45:14.516556 18063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:14.513462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34352 (local address 127.12.170.65:38935)
0926 13:45:14.513781 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:14.513792 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:14.513813 (+ 21us) server_negotiation.cc:407] Connection header received
0926 13:45:14.513861 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:14.513866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:14.514003 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:14.514421 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:14.515145 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:14.515150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:14.515157 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:45:14.515163 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:14.515188 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:45:14.515203 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:14.515306 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:14.516171 (+ 865us) server_negotiation.cc:299] Negotiation successful
0926 13:45:14.516418 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":136,"threads_started":1}
I20260926 13:45:14.516625 18062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:14.513313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46370 (local address 127.12.170.66:35773)
0926 13:45:14.514163 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:14.514173 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:14.514191 (+ 18us) server_negotiation.cc:407] Connection header received
0926 13:45:14.514236 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:14.514241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:14.514391 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:14.514515 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:14.515618 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:14.515623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:14.515632 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:45:14.515643 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:14.515702 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:45:14.515732 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:14.515820 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:14.515910 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 13:45:14.516498 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":109,"threads_started":1}
I20260926 13:45:14.518635 17981 tablet_service.cc:1467] Processing CreateTablet for tablet 05f7dc46b71c470a8f9067a6bd4f137b (DEFAULT_TABLE table=test-table [id=b10d9d5b0941448fb8190d5f18c6b4fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:14.518777 17850 tablet_service.cc:1467] Processing CreateTablet for tablet 05f7dc46b71c470a8f9067a6bd4f137b (DEFAULT_TABLE table=test-table [id=b10d9d5b0941448fb8190d5f18c6b4fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:14.519117 17719 tablet_service.cc:1467] Processing CreateTablet for tablet 05f7dc46b71c470a8f9067a6bd4f137b (DEFAULT_TABLE table=test-table [id=b10d9d5b0941448fb8190d5f18c6b4fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:14.519574 17850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05f7dc46b71c470a8f9067a6bd4f137b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:14.519611 17981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05f7dc46b71c470a8f9067a6bd4f137b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:14.520009 17719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05f7dc46b71c470a8f9067a6bd4f137b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:14.525041 18064 tablet_bootstrap.cc:492] T 05f7dc46b71c470a8f9067a6bd4f137b P ccad323169d34a36b8896cd0e3e0db48: Bootstrap starting.
I20260926 13:45:14.526539 18066 tablet_bootstrap.cc:492] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e: Bootstrap starting.
I20260926 13:45:14.527374 18064 tablet_bootstrap.cc:654] T 05f7dc46b71c470a8f9067a6bd4f137b P ccad323169d34a36b8896cd0e3e0db48: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:14.527746 18065 tablet_bootstrap.cc:492] T 05f7dc46b71c470a8f9067a6bd4f137b P 1c89af6d577f482181255aceae354972: Bootstrap starting.
I20260926 13:45:14.528240 18064 log.cc:824] T 05f7dc46b71c470a8f9067a6bd4f137b P ccad323169d34a36b8896cd0e3e0db48: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:14.528508 18066 tablet_bootstrap.cc:654] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:14.529271 18066 log.cc:824] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:14.529541 18065 tablet_bootstrap.cc:654] T 05f7dc46b71c470a8f9067a6bd4f137b P 1c89af6d577f482181255aceae354972: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:14.530231 18064 tablet_bootstrap.cc:492] T 05f7dc46b71c470a8f9067a6bd4f137b P ccad323169d34a36b8896cd0e3e0db48: No bootstrap required, opened a new log
I20260926 13:45:14.530270 18065 log.cc:824] T 05f7dc46b71c470a8f9067a6bd4f137b P 1c89af6d577f482181255aceae354972: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:14.530423 18064 ts_tablet_manager.cc:1397] T 05f7dc46b71c470a8f9067a6bd4f137b P ccad323169d34a36b8896cd0e3e0db48: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 13:45:14.530768 18066 tablet_bootstrap.cc:492] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e: No bootstrap required, opened a new log
I20260926 13:45:14.530948 18066 ts_tablet_manager.cc:1397] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 13:45:14.532068 18065 tablet_bootstrap.cc:492] T 05f7dc46b71c470a8f9067a6bd4f137b P 1c89af6d577f482181255aceae354972: No bootstrap required, opened a new log
I20260926 13:45:14.532232 18065 ts_tablet_manager.cc:1397] T 05f7dc46b71c470a8f9067a6bd4f137b P 1c89af6d577f482181255aceae354972: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 13:45:14.534269 18066 raft_consensus.cc:348] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556c21109de24aa9b1aa842df644bc1e" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 39165 } } peers { permanent_uuid: "1c89af6d577f482181255aceae354972" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 38935 } } peers { permanent_uuid: "ccad323169d34a36b8896cd0e3e0db48" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35773 } }
I20260926 13:45:14.534462 18066 raft_consensus.cc:374] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:14.534528 18066 raft_consensus.cc:729] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 556c21109de24aa9b1aa842df644bc1e, State: Initialized, Role: FOLLOWER
I20260926 13:45:14.534565 18064 raft_consensus.cc:348] T 05f7dc46b71c470a8f9067a6bd4f137b P ccad323169d34a36b8896cd0e3e0db48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556c21109de24aa9b1aa842df644bc1e" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 39165 } } peers { permanent_uuid: "1c89af6d577f482181255aceae354972" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 38935 } } peers { permanent_uuid: "ccad323169d34a36b8896cd0e3e0db48" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35773 } }
I20260926 13:45:14.534817 18064 raft_consensus.cc:374] T 05f7dc46b71c470a8f9067a6bd4f137b P ccad323169d34a36b8896cd0e3e0db48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:14.534878 18064 raft_consensus.cc:729] T 05f7dc46b71c470a8f9067a6bd4f137b P ccad323169d34a36b8896cd0e3e0db48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccad323169d34a36b8896cd0e3e0db48, State: Initialized, Role: FOLLOWER
I20260926 13:45:14.534971 18066 consensus_queue.cc:260] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556c21109de24aa9b1aa842df644bc1e" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 39165 } } peers { permanent_uuid: "1c89af6d577f482181255aceae354972" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 38935 } } peers { permanent_uuid: "ccad323169d34a36b8896cd0e3e0db48" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35773 } }
I20260926 13:45:14.535333 18064 consensus_queue.cc:260] T 05f7dc46b71c470a8f9067a6bd4f137b P ccad323169d34a36b8896cd0e3e0db48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556c21109de24aa9b1aa842df644bc1e" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 39165 } } peers { permanent_uuid: "1c89af6d577f482181255aceae354972" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 38935 } } peers { permanent_uuid: "ccad323169d34a36b8896cd0e3e0db48" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35773 } }
I20260926 13:45:14.535816 18066 ts_tablet_manager.cc:1428] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 13:45:14.535962 17916 heartbeater.cc:503] Master 127.12.170.126:44583 was elected leader, sending a full tablet report...
I20260926 13:45:14.536181 18047 heartbeater.cc:503] Master 127.12.170.126:44583 was elected leader, sending a full tablet report...
I20260926 13:45:14.536278 18064 ts_tablet_manager.cc:1428] T 05f7dc46b71c470a8f9067a6bd4f137b P ccad323169d34a36b8896cd0e3e0db48: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 13:45:14.536372 18065 raft_consensus.cc:348] T 05f7dc46b71c470a8f9067a6bd4f137b P 1c89af6d577f482181255aceae354972 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556c21109de24aa9b1aa842df644bc1e" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 39165 } } peers { permanent_uuid: "1c89af6d577f482181255aceae354972" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 38935 } } peers { permanent_uuid: "ccad323169d34a36b8896cd0e3e0db48" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35773 } }
I20260926 13:45:14.536561 18065 raft_consensus.cc:374] T 05f7dc46b71c470a8f9067a6bd4f137b P 1c89af6d577f482181255aceae354972 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:14.536603 18065 raft_consensus.cc:729] T 05f7dc46b71c470a8f9067a6bd4f137b P 1c89af6d577f482181255aceae354972 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c89af6d577f482181255aceae354972, State: Initialized, Role: FOLLOWER
I20260926 13:45:14.536931 18065 consensus_queue.cc:260] T 05f7dc46b71c470a8f9067a6bd4f137b P 1c89af6d577f482181255aceae354972 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556c21109de24aa9b1aa842df644bc1e" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 39165 } } peers { permanent_uuid: "1c89af6d577f482181255aceae354972" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 38935 } } peers { permanent_uuid: "ccad323169d34a36b8896cd0e3e0db48" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35773 } }
I20260926 13:45:14.537689 18065 ts_tablet_manager.cc:1428] T 05f7dc46b71c470a8f9067a6bd4f137b P 1c89af6d577f482181255aceae354972: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 13:45:14.538039 17785 heartbeater.cc:503] Master 127.12.170.126:44583 was elected leader, sending a full tablet report...
I20260926 13:45:14.557047 18070 raft_consensus.cc:482] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:45:14.557166 18070 raft_consensus.cc:504] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556c21109de24aa9b1aa842df644bc1e" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 39165 } } peers { permanent_uuid: "1c89af6d577f482181255aceae354972" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 38935 } } peers { permanent_uuid: "ccad323169d34a36b8896cd0e3e0db48" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35773 } }
I20260926 13:45:14.557987 18070 leader_election.cc:288] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c89af6d577f482181255aceae354972 (127.12.170.65:38935), ccad323169d34a36b8896cd0e3e0db48 (127.12.170.66:35773)
I20260926 13:45:14.559614 18049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:14.558160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:38935 (local address 127.12.170.67:48775)
0926 13:45:14.558272 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:14.558287 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:14.558346 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:14.558672 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:14.558674 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:14.558686 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:14.559129 (+ 443us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:45:14.559131 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:14.559142 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:45:14.559169 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:14.559401 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:14.559404 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:14.559408 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:45:14.559472 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 13:45:14.559505 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 13:45:14.559614 18063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:14.558229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:48775 (local address 127.12.170.65:38935)
0926 13:45:14.558351 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:14.558355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:14.558368 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:45:14.558483 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:14.558487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:14.558522 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:14.558586 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:14.559261 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:14.559265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:14.559267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:45:14.559278 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:14.559300 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:45:14.559308 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:14.559387 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:14.559482 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 13:45:14.559515 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 13:45:14.559974 18062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:14.558488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:48089 (local address 127.12.170.66:35773)
0926 13:45:14.558589 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:14.558593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:14.558797 (+ 204us) server_negotiation.cc:407] Connection header received
0926 13:45:14.558881 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:14.558884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:14.558926 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:14.558987 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:14.559578 (+ 591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:14.559581 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:14.559584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:45:14.559594 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:14.559612 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:45:14.559620 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:14.559723 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:14.559821 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 13:45:14.559859 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 13:45:14.560138 17739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f7dc46b71c470a8f9067a6bd4f137b" candidate_uuid: "556c21109de24aa9b1aa842df644bc1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c89af6d577f482181255aceae354972" is_pre_election: true
I20260926 13:45:14.560233 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:14.558392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:35773 (local address 127.12.170.67:48089)
0926 13:45:14.558708 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:14.558721 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:14.558803 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:14.559012 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:14.559015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:14.559028 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:14.559444 (+ 416us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:45:14.559446 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:14.559454 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:45:14.559477 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:14.559714 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:14.559716 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:14.559719 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:45:14.560101 (+ 382us) client_negotiation.cc:240] Negotiation successful
0926 13:45:14.560137 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":92,"threads_started":1}
I20260926 13:45:14.560379 17739 raft_consensus.cc:2454] T 05f7dc46b71c470a8f9067a6bd4f137b P 1c89af6d577f482181255aceae354972 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 556c21109de24aa9b1aa842df644bc1e in term 0.
I20260926 13:45:14.560755 17936 leader_election.cc:302] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c89af6d577f482181255aceae354972, 556c21109de24aa9b1aa842df644bc1e; no voters:
I20260926 13:45:14.560799 17870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f7dc46b71c470a8f9067a6bd4f137b" candidate_uuid: "556c21109de24aa9b1aa842df644bc1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccad323169d34a36b8896cd0e3e0db48" is_pre_election: true
I20260926 13:45:14.561000 18070 raft_consensus.cc:2781] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:45:14.561002 17870 raft_consensus.cc:2454] T 05f7dc46b71c470a8f9067a6bd4f137b P ccad323169d34a36b8896cd0e3e0db48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 556c21109de24aa9b1aa842df644bc1e in term 0.
I20260926 13:45:14.561074 18070 raft_consensus.cc:482] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:45:14.561103 18070 raft_consensus.cc:3037] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:14.562067 18070 raft_consensus.cc:504] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556c21109de24aa9b1aa842df644bc1e" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 39165 } } peers { permanent_uuid: "1c89af6d577f482181255aceae354972" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 38935 } } peers { permanent_uuid: "ccad323169d34a36b8896cd0e3e0db48" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35773 } }
I20260926 13:45:14.562356 18070 leader_election.cc:288] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e [CANDIDATE]: Term 1 election: Requested vote from peers 1c89af6d577f482181255aceae354972 (127.12.170.65:38935), ccad323169d34a36b8896cd0e3e0db48 (127.12.170.66:35773)
I20260926 13:45:14.562674 17739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f7dc46b71c470a8f9067a6bd4f137b" candidate_uuid: "556c21109de24aa9b1aa842df644bc1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c89af6d577f482181255aceae354972"
I20260926 13:45:14.562722 17870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05f7dc46b71c470a8f9067a6bd4f137b" candidate_uuid: "556c21109de24aa9b1aa842df644bc1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccad323169d34a36b8896cd0e3e0db48"
I20260926 13:45:14.562778 17739 raft_consensus.cc:3037] T 05f7dc46b71c470a8f9067a6bd4f137b P 1c89af6d577f482181255aceae354972 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:14.562815 17870 raft_consensus.cc:3037] T 05f7dc46b71c470a8f9067a6bd4f137b P ccad323169d34a36b8896cd0e3e0db48 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:14.563645 17739 raft_consensus.cc:2454] T 05f7dc46b71c470a8f9067a6bd4f137b P 1c89af6d577f482181255aceae354972 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 556c21109de24aa9b1aa842df644bc1e in term 1.
I20260926 13:45:14.563735 17870 raft_consensus.cc:2454] T 05f7dc46b71c470a8f9067a6bd4f137b P ccad323169d34a36b8896cd0e3e0db48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 556c21109de24aa9b1aa842df644bc1e in term 1.
I20260926 13:45:14.563910 17936 leader_election.cc:302] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c89af6d577f482181255aceae354972, 556c21109de24aa9b1aa842df644bc1e; no voters:
I20260926 13:45:14.564071 18070 raft_consensus.cc:2781] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:14.564252 18070 raft_consensus.cc:686] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e [term 1 LEADER]: Becoming Leader. State: Replica: 556c21109de24aa9b1aa842df644bc1e, State: Running, Role: LEADER
I20260926 13:45:14.564517 18070 consensus_queue.cc:237] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556c21109de24aa9b1aa842df644bc1e" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 39165 } } peers { permanent_uuid: "1c89af6d577f482181255aceae354972" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 38935 } } peers { permanent_uuid: "ccad323169d34a36b8896cd0e3e0db48" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35773 } }
I20260926 13:45:14.567188 17596 catalog_manager.cc:5495] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 556c21109de24aa9b1aa842df644bc1e (127.12.170.67). New cstate: current_term: 1 leader_uuid: "556c21109de24aa9b1aa842df644bc1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "556c21109de24aa9b1aa842df644bc1e" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 39165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c89af6d577f482181255aceae354972" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 38935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccad323169d34a36b8896cd0e3e0db48" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35773 } health_report { overall_health: UNKNOWN } } }
W20260926 13:45:14.573307 17917 tablet.cc:2369] T 05f7dc46b71c470a8f9067a6bd4f137b P ccad323169d34a36b8896cd0e3e0db48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:45:14.576881 18060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:14.575623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60174 (local address 127.12.170.67:39165)
0926 13:45:14.575754 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:14.575759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:14.575771 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:45:14.575910 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:14.575912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:14.575960 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:14.576024 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:14.576555 (+ 531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:14.576559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:14.576561 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:45:14.576572 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:14.576589 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:45:14.576597 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:14.576663 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:14.576742 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 13:45:14.576776 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 13:45:14.581691 17870 raft_consensus.cc:1261] T 05f7dc46b71c470a8f9067a6bd4f137b P ccad323169d34a36b8896cd0e3e0db48 [term 1 FOLLOWER]: Refusing update from remote peer 556c21109de24aa9b1aa842df644bc1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:14.581693 17739 raft_consensus.cc:1261] T 05f7dc46b71c470a8f9067a6bd4f137b P 1c89af6d577f482181255aceae354972 [term 1 FOLLOWER]: Refusing update from remote peer 556c21109de24aa9b1aa842df644bc1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:14.582257 18070 consensus_queue.cc:1035] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccad323169d34a36b8896cd0e3e0db48" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:14.582404 18074 consensus_queue.cc:1035] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c89af6d577f482181255aceae354972" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 38935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:14.589628 18079 mvcc.cc:205] Tried to move back new op lower bound from 7333602568521076736 to 7333602568458686464. Current Snapshot: MvccSnapshot[applied={T|T < 7333602568521076736}]
I20260926 13:45:14.590793 18078 mvcc.cc:205] Tried to move back new op lower bound from 7333602568521076736 to 7333602568458686464. Current Snapshot: MvccSnapshot[applied={T|T < 7333602568521076736 or (T in {7333602568521076736})}]
I20260926 13:45:14.596073 17597 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:45:14.596225 17597 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:45:14.598479 17597 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 84e1ef3a4dfe4cc88086c7aed189b64c: Sending DeleteTablet for 3 replicas of tablet 05f7dc46b71c470a8f9067a6bd4f137b
I20260926 13:45:14.599102 17981 tablet_service.cc:1514] Processing DeleteTablet for tablet 05f7dc46b71c470a8f9067a6bd4f137b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:14 UTC) from {username='slave'} at 127.0.0.1:60172
I20260926 13:45:14.599175 17719 tablet_service.cc:1514] Processing DeleteTablet for tablet 05f7dc46b71c470a8f9067a6bd4f137b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:14 UTC) from {username='slave'} at 127.0.0.1:34352
I20260926 13:45:14.599300 17850 tablet_service.cc:1514] Processing DeleteTablet for tablet 05f7dc46b71c470a8f9067a6bd4f137b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:14 UTC) from {username='slave'} at 127.0.0.1:46370
I20260926 13:45:14.599676 18088 tablet_replica.cc:330] T 05f7dc46b71c470a8f9067a6bd4f137b P ccad323169d34a36b8896cd0e3e0db48: stopping tablet replica
I20260926 13:45:14.599946 18088 raft_consensus.cc:2229] T 05f7dc46b71c470a8f9067a6bd4f137b P ccad323169d34a36b8896cd0e3e0db48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:45:14.599954 18086 tablet_replica.cc:330] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e: stopping tablet replica
I20260926 13:45:14.600029 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 17657
I20260926 13:45:14.600831 18088 raft_consensus.cc:2258] T 05f7dc46b71c470a8f9067a6bd4f137b P ccad323169d34a36b8896cd0e3e0db48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:14.600859 18086 raft_consensus.cc:2229] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:45:14.601212 18086 raft_consensus.cc:2258] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:14.602203 18088 ts_tablet_manager.cc:1905] T 05f7dc46b71c470a8f9067a6bd4f137b P ccad323169d34a36b8896cd0e3e0db48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:14.602589 18086 ts_tablet_manager.cc:1905] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:14.604517 18088 ts_tablet_manager.cc:1918] T 05f7dc46b71c470a8f9067a6bd4f137b P ccad323169d34a36b8896cd0e3e0db48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:45:14.604598 18088 log.cc:1196] T 05f7dc46b71c470a8f9067a6bd4f137b P ccad323169d34a36b8896cd0e3e0db48: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-1/wal/wals/05f7dc46b71c470a8f9067a6bd4f137b
I20260926 13:45:14.604836 18088 ts_tablet_manager.cc:1939] T 05f7dc46b71c470a8f9067a6bd4f137b P ccad323169d34a36b8896cd0e3e0db48: Deleting consensus metadata
I20260926 13:45:14.605135 18086 ts_tablet_manager.cc:1918] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:45:14.605223 18086 log.cc:1196] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/05f7dc46b71c470a8f9067a6bd4f137b
I20260926 13:45:14.605511 18086 ts_tablet_manager.cc:1939] T 05f7dc46b71c470a8f9067a6bd4f137b P 556c21109de24aa9b1aa842df644bc1e: Deleting consensus metadata
I20260926 13:45:14.605903 17583 catalog_manager.cc:4841] TS ccad323169d34a36b8896cd0e3e0db48 (127.12.170.66:35773): tablet 05f7dc46b71c470a8f9067a6bd4f137b (table test-table [id=b10d9d5b0941448fb8190d5f18c6b4fc]) successfully deleted
I20260926 13:45:14.606472 17584 catalog_manager.cc:4841] TS 556c21109de24aa9b1aa842df644bc1e (127.12.170.67:39165): tablet 05f7dc46b71c470a8f9067a6bd4f137b (table test-table [id=b10d9d5b0941448fb8190d5f18c6b4fc]) successfully deleted
W20260926 13:45:14.611032 17583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.170.65:38935 (error 108)
I20260926 13:45:14.611764 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 17788
I20260926 13:45:14.612016 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:14.611701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:38935 (local address 127.0.0.1:34366)
0926 13:45:14.611846 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:14.611941 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.65:38935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 13:45:14.613534 17583 catalog_manager.cc:4584] TS 1c89af6d577f482181255aceae354972 (127.12.170.65:38935): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 05f7dc46b71c470a8f9067a6bd4f137b: Network error: Client connection negotiation failed: client connection to 127.12.170.65:38935: connect: Connection refused (error 111)
I20260926 13:45:14.616639 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 17919
I20260926 13:45:14.620807 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 17566
2026-09-26T13:45:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (848 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 13:45:14.639919 12969 mini_oidc.cc:152] Starting JWKS server
I20260926 13:45:14.641378 12969 webserver.cc:460] Webserver started at https://127.0.0.1:35049/ using document root <none> and password file <none>
I20260926 13:45:14.641525 12969 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 13:45:14.641774 12969 webserver.cc:460] Webserver started at http://127.0.0.1:34413/ using document root <none> and password file <none>
2026-09-26T13:45:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:45:14Z Disabled control of system clock
I20260926 13:45:14.649700 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:40671
--webserver_interface=127.12.170.126
--webserver_port=0
--builtin_ntp_servers=127.12.170.84:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:40671
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35049/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790430291939677-12969-0/testchainca.cert with env {}
W20260926 13:45:14.750567 18095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:14.750792 18095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:14.750828 18095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:14.754742 18095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:45:14.754787 18095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:14.754801 18095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:14.754815 18095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:45:14.754829 18095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 13:45:14.754844 18095 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 13:45:14.754870 18095 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35049/jwks.json
I20260926 13:45:14.759884 18095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:40671
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:40671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35049/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790430291939677-12969-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:14.760762 18095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:14.761653 18095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:14.769248 18103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:14.769281 18095 server_base.cc:1034] running on GCE node
W20260926 13:45:14.769315 18101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:14.769438 18100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:14.770000 18095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:14.770933 18095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:14.772113 18095 hybrid_clock.cc:648] HybridClock initialized: now 1790430314772094 us; error 28 us; skew 500 ppm
I20260926 13:45:14.773874 18095 webserver.cc:460] Webserver started at http://127.12.170.126:34967/ using document root <none> and password file <none>
I20260926 13:45:14.774394 18095 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:14.774461 18095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:14.774650 18095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:14.776597 18095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "152a143a28b341afa8e67f450dab6fef"
format_stamp: "Formatted at 2026-09-26 13:45:14 on dist-test-slave-hc1k"
server_key: "37277adc94e1bf320c4624c6fbf882be"
server_key_iv: "2cc768c450ed349c04d2b1804367b92f"
server_key_version: "encryptionkey@0"
I20260926 13:45:14.777117 18095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "152a143a28b341afa8e67f450dab6fef"
format_stamp: "Formatted at 2026-09-26 13:45:14 on dist-test-slave-hc1k"
server_key: "37277adc94e1bf320c4624c6fbf882be"
server_key_iv: "2cc768c450ed349c04d2b1804367b92f"
server_key_version: "encryptionkey@0"
I20260926 13:45:14.780632 18095 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:45:14.782681 18109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:14.783768 18095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:45:14.783883 18095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "152a143a28b341afa8e67f450dab6fef"
format_stamp: "Formatted at 2026-09-26 13:45:14 on dist-test-slave-hc1k"
server_key: "37277adc94e1bf320c4624c6fbf882be"
server_key_iv: "2cc768c450ed349c04d2b1804367b92f"
server_key_version: "encryptionkey@0"
I20260926 13:45:14.783963 18095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:14.803134 18095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:14.803756 18095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:14.803913 18095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:14.809839 18095 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:40671
I20260926 13:45:14.809847 18161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:40671 every 8 connection(s)
I20260926 13:45:14.810634 18095 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:45:14.813452 18162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:14.814289 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 18095
I20260926 13:45:14.814419 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:45:14.814697 12969 external_mini_cluster.cc:1428] Setting key 1d0d50f6becb9518266c0eecd1d2a894
I20260926 13:45:14.818861 18162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 152a143a28b341afa8e67f450dab6fef: Bootstrap starting.
I20260926 13:45:14.820855 18162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 152a143a28b341afa8e67f450dab6fef: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:14.821527 18162 log.cc:824] T 00000000000000000000000000000000 P 152a143a28b341afa8e67f450dab6fef: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:14.822880 18165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:14.815952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47666 (local address 127.12.170.126:40671)
0926 13:45:14.816534 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:14.816547 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:14.816578 (+ 31us) server_negotiation.cc:407] Connection header received
0926 13:45:14.817176 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:14.817191 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:14.817549 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:14.817825 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:14.818217 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:14.819147 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:14.820028 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:14.820465 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:14.820599 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:14.821322 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:14.821327 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:14.821339 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:45:14.821717 (+ 378us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:14.821772 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:45:14.821791 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:14.821881 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:14.822021 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 13:45:14.822254 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":115,"threads_started":1}
I20260926 13:45:14.823222 18162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 152a143a28b341afa8e67f450dab6fef: No bootstrap required, opened a new log
I20260926 13:45:14.825246 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:45:14.826646 18162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 152a143a28b341afa8e67f450dab6fef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "152a143a28b341afa8e67f450dab6fef" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40671 } }
I20260926 13:45:14.826858 18162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 152a143a28b341afa8e67f450dab6fef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:14.826918 18162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 152a143a28b341afa8e67f450dab6fef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 152a143a28b341afa8e67f450dab6fef, State: Initialized, Role: FOLLOWER
I20260926 13:45:14.827271 18162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 152a143a28b341afa8e67f450dab6fef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "152a143a28b341afa8e67f450dab6fef" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40671 } }
I20260926 13:45:14.827378 18162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 152a143a28b341afa8e67f450dab6fef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:45:14.827443 18162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 152a143a28b341afa8e67f450dab6fef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:45:14.827530 18162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 152a143a28b341afa8e67f450dab6fef [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:14.828279 18162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 152a143a28b341afa8e67f450dab6fef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "152a143a28b341afa8e67f450dab6fef" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40671 } }
I20260926 13:45:14.828567 18162 leader_election.cc:302] T 00000000000000000000000000000000 P 152a143a28b341afa8e67f450dab6fef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 152a143a28b341afa8e67f450dab6fef; no voters:
I20260926 13:45:14.828810 18162 leader_election.cc:288] T 00000000000000000000000000000000 P 152a143a28b341afa8e67f450dab6fef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:45:14.828930 18167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 152a143a28b341afa8e67f450dab6fef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:14.829118 18167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 152a143a28b341afa8e67f450dab6fef [term 1 LEADER]: Becoming Leader. State: Replica: 152a143a28b341afa8e67f450dab6fef, State: Running, Role: LEADER
I20260926 13:45:14.829384 18167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 152a143a28b341afa8e67f450dab6fef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "152a143a28b341afa8e67f450dab6fef" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40671 } }
I20260926 13:45:14.829834 18162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 152a143a28b341afa8e67f450dab6fef [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:45:14.830716 18168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152a143a28b341afa8e67f450dab6fef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "152a143a28b341afa8e67f450dab6fef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "152a143a28b341afa8e67f450dab6fef" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40671 } } }
I20260926 13:45:14.830847 18168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152a143a28b341afa8e67f450dab6fef [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:14.831112 18169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152a143a28b341afa8e67f450dab6fef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 152a143a28b341afa8e67f450dab6fef. Latest consensus state: current_term: 1 leader_uuid: "152a143a28b341afa8e67f450dab6fef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "152a143a28b341afa8e67f450dab6fef" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40671 } } }
I20260926 13:45:14.831189 18176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:45:14.831211 18169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152a143a28b341afa8e67f450dab6fef [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:14.833828 18176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:45:14.838689 18176 catalog_manager.cc:1347] Generated new cluster ID: f460ffd187fc4783a81a49f55ea40bf9
I20260926 13:45:14.838750 18176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:45:14.850888 18176 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:45:14.851631 18176 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:45:14.873608 18176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 152a143a28b341afa8e67f450dab6fef: Generated new TSK 0
I20260926 13:45:14.874212 18176 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 13:45:14.899833 18192 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 13:45:14.899935 18192 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 13:45:14.900381 18165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:14.884049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47668 (local address 127.12.170.126:40671)
0926 13:45:14.884167 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:14.884172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:14.884354 (+ 182us) server_negotiation.cc:407] Connection header received
0926 13:45:14.884552 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:14.884554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:14.884595 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:14.884672 (+ 77us) server_negotiation.cc:226] Negotiated authn=JWT
0926 13:45:14.884908 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:14.885428 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:14.886583 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:14.886787 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:14.886853 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:14.887160 (+ 307us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 13:45:14.900104 (+ 12944us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35049/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 13:45:14.900285 (+ 181us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35049/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 13:45:14.900444 18165 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47668: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35049/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 13:45:14.901682 18191 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:14.883978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40671 (local address 127.0.0.1:47668)
0926 13:45:14.884278 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:14.884292 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:14.884423 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:14.884692 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:14.884695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:14.884702 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0926 13:45:14.884805 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:14.884810 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:14.885512 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:14.885515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:14.886487 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:14.886496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:14.886875 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:14.886879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:14.886964 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:14.887012 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 13:45:14.900283 (+ 13271us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35049/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 13:45:14.900500 (+ 217us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.170.126:40671: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35049/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":92,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.12.170.126:40671: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35049/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 13:45:14.902333 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 18095
2026-09-26T13:45:14Z chronyd exiting
I20260926 13:45:14.938620 12969 test_util.cc:175] -----------------------------------------------
I20260926 13:45:14.938694 12969 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790430291939677-12969-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (300 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 13:45:14.939630 12969 mini_oidc.cc:152] Starting JWKS server
I20260926 13:45:14.940445 12969 webserver.cc:460] Webserver started at https://127.0.0.1:34171/ using document root <none> and password file <none>
I20260926 13:45:14.940528 12969 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 13:45:14.940755 12969 webserver.cc:460] Webserver started at http://127.0.0.1:38407/ using document root <none> and password file <none>
2026-09-26T13:45:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:45:14Z Disabled control of system clock
I20260926 13:45:14.948546 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:33087
--webserver_interface=127.12.170.126
--webserver_port=0
--builtin_ntp_servers=127.12.170.84:36581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:33087
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34171/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790430291939677-12969-0/testchainca.cert with env {}
W20260926 13:45:15.049710 18198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:15.049938 18198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:15.049973 18198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:15.053843 18198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:45:15.053887 18198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:15.053907 18198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:15.053922 18198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:45:15.053937 18198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 13:45:15.053965 18198 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 13:45:15.053982 18198 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34171/jwks.json
I20260926 13:45:15.058929 18198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:36581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:33087
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:33087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34171/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790430291939677-12969-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:15.059794 18198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:15.060657 18198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:15.068135 18203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:15.068121 18204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:15.068120 18206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:15.068295 18198 server_base.cc:1034] running on GCE node
I20260926 13:45:15.068931 18198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:15.069880 18198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:15.071029 18198 hybrid_clock.cc:648] HybridClock initialized: now 1790430315071010 us; error 30 us; skew 500 ppm
I20260926 13:45:15.072577 18198 webserver.cc:460] Webserver started at http://127.12.170.126:46309/ using document root <none> and password file <none>
I20260926 13:45:15.073035 18198 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:15.073087 18198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:15.073231 18198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:15.074987 18198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "3c5c27bababf4edd9addd22aeccf2a95"
format_stamp: "Formatted at 2026-09-26 13:45:15 on dist-test-slave-hc1k"
server_key: "3ab14dc07fec1e6ddc64b3e5185f115b"
server_key_iv: "e5f5018d2972d63c26019aadd7de7d8c"
server_key_version: "encryptionkey@0"
I20260926 13:45:15.075472 18198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "3c5c27bababf4edd9addd22aeccf2a95"
format_stamp: "Formatted at 2026-09-26 13:45:15 on dist-test-slave-hc1k"
server_key: "3ab14dc07fec1e6ddc64b3e5185f115b"
server_key_iv: "e5f5018d2972d63c26019aadd7de7d8c"
server_key_version: "encryptionkey@0"
I20260926 13:45:15.078640 18198 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:45:15.080821 18212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:15.081914 18198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:15.082006 18198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "3c5c27bababf4edd9addd22aeccf2a95"
format_stamp: "Formatted at 2026-09-26 13:45:15 on dist-test-slave-hc1k"
server_key: "3ab14dc07fec1e6ddc64b3e5185f115b"
server_key_iv: "e5f5018d2972d63c26019aadd7de7d8c"
server_key_version: "encryptionkey@0"
I20260926 13:45:15.082079 18198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:15.091842 18198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:15.092347 18198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:15.092473 18198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:15.098357 18198 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:33087
I20260926 13:45:15.098367 18264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:33087 every 8 connection(s)
I20260926 13:45:15.099148 18198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:45:15.101933 18265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:15.102843 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 18198
I20260926 13:45:15.102957 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:45:15.103258 12969 external_mini_cluster.cc:1428] Setting key 109b67ea55c63447f64e99cf32753b71
I20260926 13:45:15.107411 18265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c5c27bababf4edd9addd22aeccf2a95: Bootstrap starting.
I20260926 13:45:15.109431 18265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c5c27bababf4edd9addd22aeccf2a95: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:15.110240 18265 log.cc:824] T 00000000000000000000000000000000 P 3c5c27bababf4edd9addd22aeccf2a95: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:15.111449 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:15.104392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60472 (local address 127.12.170.126:33087)
0926 13:45:15.104936 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:15.104950 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:15.104981 (+ 31us) server_negotiation.cc:407] Connection header received
0926 13:45:15.105565 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:15.105581 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:15.105946 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:15.106251 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:15.106565 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:15.107451 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:15.108320 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:15.108757 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:15.108893 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:15.109665 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:15.109671 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:15.109687 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:45:15.110135 (+ 448us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:15.110203 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:45:15.110227 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:15.110322 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:15.110436 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 13:45:15.110665 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":95,"threads_started":1}
I20260926 13:45:15.112110 18265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c5c27bababf4edd9addd22aeccf2a95: No bootstrap required, opened a new log
I20260926 13:45:15.113814 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:45:15.115538 18265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c5c27bababf4edd9addd22aeccf2a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5c27bababf4edd9addd22aeccf2a95" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33087 } }
I20260926 13:45:15.115756 18265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c5c27bababf4edd9addd22aeccf2a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:15.115795 18265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c5c27bababf4edd9addd22aeccf2a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5c27bababf4edd9addd22aeccf2a95, State: Initialized, Role: FOLLOWER
I20260926 13:45:15.116132 18265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c5c27bababf4edd9addd22aeccf2a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5c27bababf4edd9addd22aeccf2a95" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33087 } }
I20260926 13:45:15.116242 18265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c5c27bababf4edd9addd22aeccf2a95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:45:15.116283 18265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c5c27bababf4edd9addd22aeccf2a95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:45:15.116346 18265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c5c27bababf4edd9addd22aeccf2a95 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:15.117044 18265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c5c27bababf4edd9addd22aeccf2a95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5c27bababf4edd9addd22aeccf2a95" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33087 } }
I20260926 13:45:15.117302 18265 leader_election.cc:302] T 00000000000000000000000000000000 P 3c5c27bababf4edd9addd22aeccf2a95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c5c27bababf4edd9addd22aeccf2a95; no voters:
I20260926 13:45:15.117527 18265 leader_election.cc:288] T 00000000000000000000000000000000 P 3c5c27bababf4edd9addd22aeccf2a95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:45:15.117633 18270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c5c27bababf4edd9addd22aeccf2a95 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:15.117816 18270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c5c27bababf4edd9addd22aeccf2a95 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5c27bababf4edd9addd22aeccf2a95, State: Running, Role: LEADER
I20260926 13:45:15.118063 18270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c5c27bababf4edd9addd22aeccf2a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5c27bababf4edd9addd22aeccf2a95" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33087 } }
I20260926 13:45:15.118532 18265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c5c27bababf4edd9addd22aeccf2a95 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:45:15.119501 18272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c5c27bababf4edd9addd22aeccf2a95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c5c27bababf4edd9addd22aeccf2a95. Latest consensus state: current_term: 1 leader_uuid: "3c5c27bababf4edd9addd22aeccf2a95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5c27bababf4edd9addd22aeccf2a95" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33087 } } }
I20260926 13:45:15.119551 18271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c5c27bababf4edd9addd22aeccf2a95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c5c27bababf4edd9addd22aeccf2a95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5c27bababf4edd9addd22aeccf2a95" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33087 } } }
I20260926 13:45:15.119614 18272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c5c27bababf4edd9addd22aeccf2a95 [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:15.119663 18271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c5c27bababf4edd9addd22aeccf2a95 [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:15.119935 18279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:45:15.122430 18279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:45:15.127285 18279 catalog_manager.cc:1347] Generated new cluster ID: f5268414f8684f319585a7249554b398
I20260926 13:45:15.127341 18279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:45:15.146838 18279 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:45:15.148010 18279 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:45:15.159108 18279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c5c27bababf4edd9addd22aeccf2a95: Generated new TSK 0
I20260926 13:45:15.159852 18279 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 13:45:15.184114 18295 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 13:45:15.184172 18295 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 13:45:15.184813 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:15.169789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60480 (local address 127.12.170.126:33087)
0926 13:45:15.169897 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:15.169901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:15.170045 (+ 144us) server_negotiation.cc:407] Connection header received
0926 13:45:15.170162 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:15.170164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:15.170218 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:15.170339 (+ 121us) server_negotiation.cc:226] Negotiated authn=JWT
0926 13:45:15.170513 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:15.171316 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:15.172129 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:15.172476 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:15.172574 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:15.172782 (+ 208us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 13:45:15.184440 (+ 11658us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 13:45:15.184679 (+ 239us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60480: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 13:45:15.184844 18294 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:15.169703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33087 (local address 127.0.0.1:60480)
0926 13:45:15.169975 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:15.169986 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:15.170072 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:15.170329 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:15.170331 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:15.170337 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0926 13:45:15.170432 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:15.170437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:15.171407 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:15.171409 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:15.172036 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:15.172045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:15.172574 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:15.172577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:15.172633 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:15.172653 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 13:45:15.184597 (+ 11944us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 13:45:15.184722 (+ 125us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.170.126:33087: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":92,"threads_started":1}
W20260926 13:45:15.184868 18268 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60480: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260926 13:45:15.185568 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 18198
2026-09-26T13:45:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (283 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 13:45:15.222986 12969 mini_oidc.cc:152] Starting JWKS server
I20260926 13:45:15.223811 12969 webserver.cc:460] Webserver started at https://127.0.0.1:42265/ using document root <none> and password file <none>
I20260926 13:45:15.223904 12969 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 13:45:15.224135 12969 webserver.cc:460] Webserver started at http://127.0.0.1:33761/ using document root <none> and password file <none>
2026-09-26T13:45:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:45:15Z Disabled control of system clock
I20260926 13:45:15.231889 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:43429
--webserver_interface=127.12.170.126
--webserver_port=0
--builtin_ntp_servers=127.12.170.84:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:43429
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42265/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 13:45:15.332676 18301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:15.332892 18301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:15.332926 18301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:15.336705 18301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:45:15.336746 18301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:15.336759 18301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:15.336773 18301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:45:15.336787 18301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 13:45:15.336800 18301 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 13:45:15.336815 18301 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42265/jwks.json
I20260926 13:45:15.341228 18301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:43429
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:43429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42265/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:15.342062 18301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:15.343192 18301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:15.350596 18307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:15.350593 18309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:15.350757 18306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:15.350817 18301 server_base.cc:1034] running on GCE node
I20260926 13:45:15.351492 18301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:15.352502 18301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:15.353675 18301 hybrid_clock.cc:648] HybridClock initialized: now 1790430315353657 us; error 29 us; skew 500 ppm
I20260926 13:45:15.355479 18301 webserver.cc:460] Webserver started at http://127.12.170.126:35237/ using document root <none> and password file <none>
I20260926 13:45:15.356045 18301 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:15.356114 18301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:15.356312 18301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:15.358210 18301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "7b413281c68c47818cdee7da4790fb36"
format_stamp: "Formatted at 2026-09-26 13:45:15 on dist-test-slave-hc1k"
server_key: "5e290cb837df229e4598f16cb2f9d69b"
server_key_iv: "f9292dec3ed9cef1acb0003760ddb552"
server_key_version: "encryptionkey@0"
I20260926 13:45:15.358736 18301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "7b413281c68c47818cdee7da4790fb36"
format_stamp: "Formatted at 2026-09-26 13:45:15 on dist-test-slave-hc1k"
server_key: "5e290cb837df229e4598f16cb2f9d69b"
server_key_iv: "f9292dec3ed9cef1acb0003760ddb552"
server_key_version: "encryptionkey@0"
I20260926 13:45:15.362215 18301 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 13:45:15.364588 18315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:15.365700 18301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:45:15.365804 18301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "7b413281c68c47818cdee7da4790fb36"
format_stamp: "Formatted at 2026-09-26 13:45:15 on dist-test-slave-hc1k"
server_key: "5e290cb837df229e4598f16cb2f9d69b"
server_key_iv: "f9292dec3ed9cef1acb0003760ddb552"
server_key_version: "encryptionkey@0"
I20260926 13:45:15.365907 18301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:15.397615 18301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:15.398228 18301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:15.398388 18301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:15.405136 18301 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:43429
I20260926 13:45:15.405148 18367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:43429 every 8 connection(s)
I20260926 13:45:15.406062 18301 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:45:15.406416 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 18301
I20260926 13:45:15.406523 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:45:15.406800 12969 external_mini_cluster.cc:1428] Setting key 740326921df508b46fb2db4698d3fcb1
I20260926 13:45:15.409152 18368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:15.415216 18368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b413281c68c47818cdee7da4790fb36: Bootstrap starting.
I20260926 13:45:15.415544 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:15.408107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56982 (local address 127.12.170.126:43429)
0926 13:45:15.408665 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:15.408681 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:15.408713 (+ 32us) server_negotiation.cc:407] Connection header received
0926 13:45:15.409341 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:15.409360 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:15.409726 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:15.409998 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:15.410382 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:15.411307 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:15.412534 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:15.412978 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:15.413094 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:15.413966 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:15.413971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:15.413982 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:45:15.414318 (+ 336us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:15.414382 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:45:15.414402 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:15.414473 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:15.414635 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 13:45:15.414863 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":108,"threads_started":1}
I20260926 13:45:15.417593 18368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b413281c68c47818cdee7da4790fb36: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:15.418103 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:45:15.418468 18368 log.cc:824] T 00000000000000000000000000000000 P 7b413281c68c47818cdee7da4790fb36: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:15.420362 18368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b413281c68c47818cdee7da4790fb36: No bootstrap required, opened a new log
I20260926 13:45:15.423480 18368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b413281c68c47818cdee7da4790fb36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b413281c68c47818cdee7da4790fb36" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 43429 } }
I20260926 13:45:15.423691 18368 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b413281c68c47818cdee7da4790fb36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:15.423741 18368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b413281c68c47818cdee7da4790fb36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b413281c68c47818cdee7da4790fb36, State: Initialized, Role: FOLLOWER
I20260926 13:45:15.424108 18368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b413281c68c47818cdee7da4790fb36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b413281c68c47818cdee7da4790fb36" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 43429 } }
I20260926 13:45:15.424221 18368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b413281c68c47818cdee7da4790fb36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:45:15.424258 18368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b413281c68c47818cdee7da4790fb36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:45:15.424324 18368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b413281c68c47818cdee7da4790fb36 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:15.425086 18368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b413281c68c47818cdee7da4790fb36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b413281c68c47818cdee7da4790fb36" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 43429 } }
I20260926 13:45:15.425387 18368 leader_election.cc:302] T 00000000000000000000000000000000 P 7b413281c68c47818cdee7da4790fb36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b413281c68c47818cdee7da4790fb36; no voters:
I20260926 13:45:15.425669 18368 leader_election.cc:288] T 00000000000000000000000000000000 P 7b413281c68c47818cdee7da4790fb36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:45:15.425755 18373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b413281c68c47818cdee7da4790fb36 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:15.425990 18373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b413281c68c47818cdee7da4790fb36 [term 1 LEADER]: Becoming Leader. State: Replica: 7b413281c68c47818cdee7da4790fb36, State: Running, Role: LEADER
I20260926 13:45:15.426251 18373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b413281c68c47818cdee7da4790fb36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b413281c68c47818cdee7da4790fb36" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 43429 } }
I20260926 13:45:15.426694 18368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b413281c68c47818cdee7da4790fb36 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:45:15.427640 18375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b413281c68c47818cdee7da4790fb36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b413281c68c47818cdee7da4790fb36. Latest consensus state: current_term: 1 leader_uuid: "7b413281c68c47818cdee7da4790fb36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b413281c68c47818cdee7da4790fb36" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 43429 } } }
I20260926 13:45:15.427796 18375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b413281c68c47818cdee7da4790fb36 [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:15.428077 18374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b413281c68c47818cdee7da4790fb36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b413281c68c47818cdee7da4790fb36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b413281c68c47818cdee7da4790fb36" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 43429 } } }
I20260926 13:45:15.428153 18382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:45:15.428169 18374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b413281c68c47818cdee7da4790fb36 [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:15.431077 18382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:45:15.435678 18382 catalog_manager.cc:1347] Generated new cluster ID: 1078ff84a3a341a8811194701fd20a71
I20260926 13:45:15.435731 18382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:45:15.453826 18382 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:45:15.454589 18382 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:45:15.461937 18382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b413281c68c47818cdee7da4790fb36: Generated new TSK 0
I20260926 13:45:15.462484 18382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:45:15.480209 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:15.475574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56984 (local address 127.12.170.126:43429)
0926 13:45:15.475704 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:15.475708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:15.475894 (+ 186us) server_negotiation.cc:407] Connection header received
0926 13:45:15.476050 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:15.476052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:15.476095 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:15.476175 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0926 13:45:15.476394 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:15.476895 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:15.477721 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:15.477935 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:15.478015 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:15.478281 (+ 266us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 13:45:15.479895 (+ 1614us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42265/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 13:45:15.480091 (+ 196us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56984: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42265/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 13:45:15.480273 18371 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56984: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42265/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 13:45:15.480465 18397 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:15.475529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:43429 (local address 127.0.0.1:56984)
0926 13:45:15.475810 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:15.475821 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:15.475927 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:15.476182 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:15.476185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:15.476193 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0926 13:45:15.476301 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:15.476307 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:15.476995 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:15.476999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:15.477618 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:15.477625 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:15.478033 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:15.478037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:15.478123 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:15.478144 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 13:45:15.480027 (+ 1883us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42265/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 13:45:15.480153 (+ 126us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.170.126:43429: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42265/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":123,"threads_started":1}
I20260926 13:45:15.481061 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 18301
E20260926 13:45:15.493196 18398 webserver.cc:533] Webserver: sslize failed:
E20260926 13:45:15.493254 18398 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T13:45:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (297 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T13:45:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:45:15Z Disabled control of system clock
I20260926 13:45:15.527482 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:45035
--webserver_interface=127.12.170.126
--webserver_port=0
--builtin_ntp_servers=127.12.170.84:41871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:45035
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790430291939677-12969-0/testchainca.cert with env {}
W20260926 13:45:15.629192 18402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:15.629418 18402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:15.629453 18402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:15.633236 18402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:45:15.633276 18402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:15.633289 18402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:15.633303 18402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:45:15.633316 18402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 13:45:15.633335 18402 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 13:45:15.633359 18402 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 13:45:15.638264 18402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:41871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:45035
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:45035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790430291939677-12969-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:15.639081 18402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:15.639926 18402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:15.647290 18410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:15.647360 18407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:15.647500 18402 server_base.cc:1034] running on GCE node
W20260926 13:45:15.647559 18408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:15.648059 18402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:15.648973 18402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:15.650118 18402 hybrid_clock.cc:648] HybridClock initialized: now 1790430315650108 us; error 23 us; skew 500 ppm
I20260926 13:45:15.651749 18402 webserver.cc:460] Webserver started at http://127.12.170.126:36285/ using document root <none> and password file <none>
I20260926 13:45:15.652231 18402 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:15.652285 18402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:15.652468 18402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:15.654218 18402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "bd4c04a3d52d45ffbf64ece488d5413f"
format_stamp: "Formatted at 2026-09-26 13:45:15 on dist-test-slave-hc1k"
server_key: "49de6b6e9c5f2c37bfdb60e329aae793"
server_key_iv: "8eb52ee80b949840defc0049d71372d5"
server_key_version: "encryptionkey@0"
I20260926 13:45:15.654712 18402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "bd4c04a3d52d45ffbf64ece488d5413f"
format_stamp: "Formatted at 2026-09-26 13:45:15 on dist-test-slave-hc1k"
server_key: "49de6b6e9c5f2c37bfdb60e329aae793"
server_key_iv: "8eb52ee80b949840defc0049d71372d5"
server_key_version: "encryptionkey@0"
I20260926 13:45:15.657938 18402 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 13:45:15.660164 18416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:15.661279 18402 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:15.661396 18402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "bd4c04a3d52d45ffbf64ece488d5413f"
format_stamp: "Formatted at 2026-09-26 13:45:15 on dist-test-slave-hc1k"
server_key: "49de6b6e9c5f2c37bfdb60e329aae793"
server_key_iv: "8eb52ee80b949840defc0049d71372d5"
server_key_version: "encryptionkey@0"
I20260926 13:45:15.661494 18402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:15.693707 18402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:15.694327 18402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:15.694484 18402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:15.700769 18402 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:45035
I20260926 13:45:15.700868 18468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:45035 every 8 connection(s)
I20260926 13:45:15.701639 18402 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:45:15.701974 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 18402
I20260926 13:45:15.702090 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:45:15.702426 12969 external_mini_cluster.cc:1428] Setting key 63f44144b675061d95f14ac90380cdb9
I20260926 13:45:15.704728 18469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:15.710438 18469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd4c04a3d52d45ffbf64ece488d5413f: Bootstrap starting.
I20260926 13:45:15.710952 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:15.703778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44680 (local address 127.12.170.126:45035)
0926 13:45:15.704353 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:15.704371 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:15.704404 (+ 33us) server_negotiation.cc:407] Connection header received
0926 13:45:15.705048 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:15.705065 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:15.705435 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:15.705703 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:15.706077 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:15.706986 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:15.707922 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:15.708372 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:15.708510 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:15.709336 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:15.709340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:15.709350 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:45:15.709708 (+ 358us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:15.709758 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:45:15.709777 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:15.709856 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:15.709993 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 13:45:15.710214 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":116,"threads_started":1}
I20260926 13:45:15.712662 18469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd4c04a3d52d45ffbf64ece488d5413f: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:15.713344 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:45:15.713402 18469 log.cc:824] T 00000000000000000000000000000000 P bd4c04a3d52d45ffbf64ece488d5413f: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:15.715279 18469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd4c04a3d52d45ffbf64ece488d5413f: No bootstrap required, opened a new log
I20260926 13:45:15.718461 18469 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd4c04a3d52d45ffbf64ece488d5413f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4c04a3d52d45ffbf64ece488d5413f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45035 } }
I20260926 13:45:15.718654 18469 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd4c04a3d52d45ffbf64ece488d5413f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:15.718686 18469 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd4c04a3d52d45ffbf64ece488d5413f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd4c04a3d52d45ffbf64ece488d5413f, State: Initialized, Role: FOLLOWER
I20260926 13:45:15.719044 18469 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd4c04a3d52d45ffbf64ece488d5413f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4c04a3d52d45ffbf64ece488d5413f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45035 } }
I20260926 13:45:15.719180 18469 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd4c04a3d52d45ffbf64ece488d5413f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:45:15.719245 18469 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd4c04a3d52d45ffbf64ece488d5413f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:45:15.719344 18469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd4c04a3d52d45ffbf64ece488d5413f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:15.720189 18469 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd4c04a3d52d45ffbf64ece488d5413f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4c04a3d52d45ffbf64ece488d5413f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45035 } }
I20260926 13:45:15.720472 18469 leader_election.cc:302] T 00000000000000000000000000000000 P bd4c04a3d52d45ffbf64ece488d5413f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd4c04a3d52d45ffbf64ece488d5413f; no voters:
I20260926 13:45:15.720695 18469 leader_election.cc:288] T 00000000000000000000000000000000 P bd4c04a3d52d45ffbf64ece488d5413f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:45:15.720825 18474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd4c04a3d52d45ffbf64ece488d5413f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:15.721101 18474 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd4c04a3d52d45ffbf64ece488d5413f [term 1 LEADER]: Becoming Leader. State: Replica: bd4c04a3d52d45ffbf64ece488d5413f, State: Running, Role: LEADER
I20260926 13:45:15.721357 18474 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd4c04a3d52d45ffbf64ece488d5413f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4c04a3d52d45ffbf64ece488d5413f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45035 } }
I20260926 13:45:15.721715 18469 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd4c04a3d52d45ffbf64ece488d5413f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:45:15.722735 18475 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd4c04a3d52d45ffbf64ece488d5413f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd4c04a3d52d45ffbf64ece488d5413f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4c04a3d52d45ffbf64ece488d5413f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45035 } } }
I20260926 13:45:15.722831 18475 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd4c04a3d52d45ffbf64ece488d5413f [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:15.723100 18476 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd4c04a3d52d45ffbf64ece488d5413f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd4c04a3d52d45ffbf64ece488d5413f. Latest consensus state: current_term: 1 leader_uuid: "bd4c04a3d52d45ffbf64ece488d5413f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd4c04a3d52d45ffbf64ece488d5413f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45035 } } }
I20260926 13:45:15.723183 18476 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd4c04a3d52d45ffbf64ece488d5413f [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:15.723318 18483 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:45:15.726429 18483 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:45:15.730898 18483 catalog_manager.cc:1347] Generated new cluster ID: c908eabf6d7c4890b179a463761f739c
I20260926 13:45:15.730954 18483 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:45:15.745726 18483 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:45:15.746531 18483 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:45:15.752707 18483 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd4c04a3d52d45ffbf64ece488d5413f: Generated new TSK 0
I20260926 13:45:15.753255 18483 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:45:15.839709 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:15.765370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44684 (local address 127.12.170.126:45035)
0926 13:45:15.765488 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:15.765492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:15.765506 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:45:15.765571 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:15.765574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:15.765623 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:15.765718 (+ 95us) server_negotiation.cc:226] Negotiated authn=JWT
0926 13:45:15.766005 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:15.766796 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:15.767641 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:15.768009 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:15.768106 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:15.768392 (+ 286us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 13:45:15.839220 (+ 70828us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 13:45:15.839527 (+ 307us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44684: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 13:45:15.839789 18472 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44684: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 13:45:15.839823 18498 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:15.765038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:45035 (local address 127.0.0.1:44684)
0926 13:45:15.765329 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:15.765344 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:15.765455 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:15.765729 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:15.765731 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:15.765737 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0926 13:45:15.765903 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:15.765910 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:15.766898 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:15.766900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:15.767542 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:15.767550 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:15.768142 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:15.768146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:15.768222 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:15.768246 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 13:45:15.839446 (+ 71200us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 13:45:15.839620 (+ 174us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.170.126:45035: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":104,"threads_started":1}
I20260926 13:45:15.840581 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 18402
2026-09-26T13:45:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (338 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (116 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (111 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:45:16 dist-test-slave-hc1k krb5kdc[18510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:45:16 dist-test-slave-hc1k krb5kdc[18510](info): set up 2 sockets
Sep 26 13:45:16 dist-test-slave-hc1k krb5kdc[18510](info): commencing operation
krb5kdc: starting...
W20260926 13:45:17.019524 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:45:17 dist-test-slave-hc1k krb5kdc[18510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:45:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:45:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:45:17.126521 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:34021
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:40361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:34021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:17.230201 18523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:17.230428 18523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:17.230465 18523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:17.234342 18523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:45:17.234387 18523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:17.234404 18523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:17.234419 18523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:45:17.234433 18523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:45:17.239338 18523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:40361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:34021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:34021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:17.240226 18523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:17.241139 18523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:17.248771 18528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:17.248716 18529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:17.248757 18523 server_base.cc:1034] running on GCE node
W20260926 13:45:17.248715 18531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:17.249488 18523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:17.250412 18523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:17.251591 18523 hybrid_clock.cc:648] HybridClock initialized: now 1790430317251576 us; error 25 us; skew 500 ppm
Sep 26 13:45:17 dist-test-slave-hc1k krb5kdc[18510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430317, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:17.254143 18523 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:45:17.254933 18523 webserver.cc:460] Webserver started at http://127.12.170.126:34387/ using document root <none> and password file <none>
I20260926 13:45:17.255415 18523 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:17.255477 18523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:17.255673 18523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:17.257561 18523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "fbd5c284f4e94f3c90cd5bc917f4473c"
format_stamp: "Formatted at 2026-09-26 13:45:17 on dist-test-slave-hc1k"
server_key: "2d2cb3a7e2df1e1a999e80d6919f65f3"
server_key_iv: "b657fd8afa3a97802bbee34937a99be4"
server_key_version: "encryptionkey@0"
I20260926 13:45:17.258069 18523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "fbd5c284f4e94f3c90cd5bc917f4473c"
format_stamp: "Formatted at 2026-09-26 13:45:17 on dist-test-slave-hc1k"
server_key: "2d2cb3a7e2df1e1a999e80d6919f65f3"
server_key_iv: "b657fd8afa3a97802bbee34937a99be4"
server_key_version: "encryptionkey@0"
I20260926 13:45:17.261596 18523 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:45:17.263787 18538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:17.264919 18523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:45:17.265036 18523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "fbd5c284f4e94f3c90cd5bc917f4473c"
format_stamp: "Formatted at 2026-09-26 13:45:17 on dist-test-slave-hc1k"
server_key: "2d2cb3a7e2df1e1a999e80d6919f65f3"
server_key_iv: "b657fd8afa3a97802bbee34937a99be4"
server_key_version: "encryptionkey@0"
I20260926 13:45:17.265120 18523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:17.280334 18523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:17.282814 18523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:17.282968 18523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:17.289203 18523 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:34021
I20260926 13:45:17.289237 18590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:34021 every 8 connection(s)
I20260926 13:45:17.290050 18523 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:45:17.291100 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 18523
I20260926 13:45:17.291211 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:45:17.291464 12969 external_mini_cluster.cc:1428] Setting key 0706998dc8f53430b3b4aafcbbb54fd9
I20260926 13:45:17.292943 18591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:17.299443 18591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbd5c284f4e94f3c90cd5bc917f4473c: Bootstrap starting.
Sep 26 13:45:17 dist-test-slave-hc1k krb5kdc[18510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:17.301955 18591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbd5c284f4e94f3c90cd5bc917f4473c: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:17.302812 18591 log.cc:824] T 00000000000000000000000000000000 P fbd5c284f4e94f3c90cd5bc917f4473c: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:17.304975 18591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbd5c284f4e94f3c90cd5bc917f4473c: No bootstrap required, opened a new log
I20260926 13:45:17.305943 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:17.292709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50080 (local address 127.12.170.126:34021)
0926 13:45:17.293101 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:17.293121 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:17.293161 (+ 40us) server_negotiation.cc:407] Connection header received
0926 13:45:17.294003 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:17.294023 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:17.294416 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:17.294764 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:17.295397 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:17.296872 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:17.297701 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:17.298392 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:17.298556 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:17.301083 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:17.301090 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:17.301104 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:17.301136 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:17.303035 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:17.303492 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:17.303496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:17.303500 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:17.303643 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:17.303931 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:17.303934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:17.303936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:17.304409 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:17.304606 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:17.304773 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 13:45:17.305060 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":113,"threads_started":1}
I20260926 13:45:17.308823 18591 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbd5c284f4e94f3c90cd5bc917f4473c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbd5c284f4e94f3c90cd5bc917f4473c" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34021 } }
I20260926 13:45:17.309026 18591 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbd5c284f4e94f3c90cd5bc917f4473c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:17.309077 18591 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbd5c284f4e94f3c90cd5bc917f4473c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbd5c284f4e94f3c90cd5bc917f4473c, State: Initialized, Role: FOLLOWER
I20260926 13:45:17.309458 18591 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbd5c284f4e94f3c90cd5bc917f4473c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbd5c284f4e94f3c90cd5bc917f4473c" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34021 } }
I20260926 13:45:17.309581 18591 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbd5c284f4e94f3c90cd5bc917f4473c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:45:17.309629 18591 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbd5c284f4e94f3c90cd5bc917f4473c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:45:17.309702 18591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbd5c284f4e94f3c90cd5bc917f4473c [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:17.310526 18591 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbd5c284f4e94f3c90cd5bc917f4473c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbd5c284f4e94f3c90cd5bc917f4473c" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34021 } }
I20260926 13:45:17.310806 18591 leader_election.cc:302] T 00000000000000000000000000000000 P fbd5c284f4e94f3c90cd5bc917f4473c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbd5c284f4e94f3c90cd5bc917f4473c; no voters:
I20260926 13:45:17.311043 18591 leader_election.cc:288] T 00000000000000000000000000000000 P fbd5c284f4e94f3c90cd5bc917f4473c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:45:17.311141 18596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbd5c284f4e94f3c90cd5bc917f4473c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:17.311334 18596 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbd5c284f4e94f3c90cd5bc917f4473c [term 1 LEADER]: Becoming Leader. State: Replica: fbd5c284f4e94f3c90cd5bc917f4473c, State: Running, Role: LEADER
I20260926 13:45:17.311630 18596 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbd5c284f4e94f3c90cd5bc917f4473c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbd5c284f4e94f3c90cd5bc917f4473c" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34021 } }
I20260926 13:45:17.312093 18591 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbd5c284f4e94f3c90cd5bc917f4473c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:45:17.313133 18598 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbd5c284f4e94f3c90cd5bc917f4473c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbd5c284f4e94f3c90cd5bc917f4473c. Latest consensus state: current_term: 1 leader_uuid: "fbd5c284f4e94f3c90cd5bc917f4473c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbd5c284f4e94f3c90cd5bc917f4473c" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34021 } } }
I20260926 13:45:17.313259 18598 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbd5c284f4e94f3c90cd5bc917f4473c [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:17.313611 18597 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbd5c284f4e94f3c90cd5bc917f4473c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbd5c284f4e94f3c90cd5bc917f4473c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbd5c284f4e94f3c90cd5bc917f4473c" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34021 } } }
I20260926 13:45:17.313766 18597 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbd5c284f4e94f3c90cd5bc917f4473c [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:17.314062 18605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:45:17.316570 18605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:45:17.321223 18605 catalog_manager.cc:1347] Generated new cluster ID: 413b4951d63947f09a43a388ce448bbc
I20260926 13:45:17.321287 18605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:45:17.329149 18605 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:45:17.329910 18605 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:45:17.333181 18605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbd5c284f4e94f3c90cd5bc917f4473c: Generated new TSK 0
I20260926 13:45:17.333707 18605 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.170.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
I20260926 13:45:17.364346 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:0
--local_ip_for_outbound_sockets=127.12.170.65
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:34021
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:40361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:17.501914 18617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:17.502140 18617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:17.502182 18617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:17.506178 18617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:17.506237 18617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:17.506330 18617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:45:17.511447 18617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:40361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:34021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:17.512414 18617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:17.513440 18617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:17.522135 18625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:17.522094 18622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:17.522106 18623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:17.522305 18617 server_base.cc:1034] running on GCE node
I20260926 13:45:17.522651 18617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:17.523195 18617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:17.524358 18617 hybrid_clock.cc:648] HybridClock initialized: now 1790430317524336 us; error 31 us; skew 500 ppm
Sep 26 13:45:17 dist-test-slave-hc1k krb5kdc[18510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430317, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:17.526901 18617 init.cc:377] Logged in from keytab as kudu/127.12.170.65@KRBTEST.COM (short username kudu)
I20260926 13:45:17.527702 18617 webserver.cc:460] Webserver started at http://127.12.170.65:37889/ using document root <none> and password file <none>
I20260926 13:45:17.528180 18617 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:17.528236 18617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:17.528396 18617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:17.530263 18617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-0/data/instance:
uuid: "17587251bd6a49ec996588214d7e0cb9"
format_stamp: "Formatted at 2026-09-26 13:45:17 on dist-test-slave-hc1k"
server_key: "c530e952e23d2f650f24771eb6aa7d15"
server_key_iv: "b4df24936470730566907d6f898307da"
server_key_version: "encryptionkey@0"
I20260926 13:45:17.530756 18617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance:
uuid: "17587251bd6a49ec996588214d7e0cb9"
format_stamp: "Formatted at 2026-09-26 13:45:17 on dist-test-slave-hc1k"
server_key: "c530e952e23d2f650f24771eb6aa7d15"
server_key_iv: "b4df24936470730566907d6f898307da"
server_key_version: "encryptionkey@0"
I20260926 13:45:17.534229 18617 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 13:45:17.536614 18632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:17.537676 18617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:45:17.537778 18617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "17587251bd6a49ec996588214d7e0cb9"
format_stamp: "Formatted at 2026-09-26 13:45:17 on dist-test-slave-hc1k"
server_key: "c530e952e23d2f650f24771eb6aa7d15"
server_key_iv: "b4df24936470730566907d6f898307da"
server_key_version: "encryptionkey@0"
I20260926 13:45:17.537899 18617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:17.547062 18617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:17.549434 18617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:17.549576 18617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:17.550088 18617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:17.550786 18617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:17.550837 18617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:17.550889 18617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:17.550915 18617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:17.558687 18617 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:46487
I20260926 13:45:17.558714 18745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:46487 every 8 connection(s)
I20260926 13:45:17.559430 18617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:45:17.560149 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 18617
I20260926 13:45:17.560263 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance
I20260926 13:45:17.560560 12969 external_mini_cluster.cc:1428] Setting key ef1ac378c817054f250e5d349c80573f
Sep 26 13:45:17 dist-test-slave-hc1k krb5kdc[18510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430317, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.170.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.66@KRBTEST.COM" created.
I20260926 13:45:17.572644 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:17.561102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:46653 (local address 127.12.170.126:34021)
0926 13:45:17.561215 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:17.561219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:17.562048 (+ 829us) server_negotiation.cc:407] Connection header received
0926 13:45:17.563303 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:17.563305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:17.563356 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:17.563419 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:17.564763 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:17.565277 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:17.566862 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:17.567091 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:17.567169 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:17.569501 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:17.569505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:17.569506 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:17.569529 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:17.570693 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:17.571184 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:17.571187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:17.571189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:17.571300 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:17.571633 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:17.571636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:17.571638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:17.571836 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:17.571920 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:17.572370 (+ 450us) server_negotiation.cc:299] Negotiation successful
0926 13:45:17.572486 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 13:45:17.573326 18748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:17.561454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34021 (local address 127.12.170.65:46653)
0926 13:45:17.561893 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:17.561946 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:17.563098 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:17.563591 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:17.563599 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:17.564042 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:17.564631 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:17.564648 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:17.565393 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:17.565397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:17.566738 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:17.566749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:17.567234 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:17.567238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:17.567422 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:17.568031 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:17.568046 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:17.569359 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:17.570799 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:17.570805 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:17.570827 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:17.571097 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:17.571391 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:17.571394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:17.571396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:17.571550 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:17.571934 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:17.571940 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:17.572154 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 13:45:17.572308 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 13:45:17.572586 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":128,"threads_started":1}
I20260926 13:45:17.574366 18746 heartbeater.cc:347] Connected to a master server at 127.12.170.126:34021
I20260926 13:45:17.574561 18746 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:17.574968 18746 heartbeater.cc:511] Master 127.12.170.126:34021 requested a full tablet report, sending...
I20260926 13:45:17.576489 18555 ts_manager.cc:194] Registered new tserver with Master: 17587251bd6a49ec996588214d7e0cb9 (127.12.170.65:46487)
I20260926 13:45:17.578017 18555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.65@KRBTEST.COM'} at 127.12.170.65:46653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
I20260926 13:45:17.583756 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:0
--local_ip_for_outbound_sockets=127.12.170.66
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:34021
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:40361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:17.684669 18751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:17.684880 18751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:17.684917 18751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:17.688777 18751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:17.688822 18751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:17.688874 18751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:45:17.693693 18751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:40361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:34021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:17.694506 18751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:17.695377 18751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:17.703770 18757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:17.703778 18756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:17.703850 18751 server_base.cc:1034] running on GCE node
W20260926 13:45:17.703778 18759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:17.704325 18751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:17.704840 18751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:17.705986 18751 hybrid_clock.cc:648] HybridClock initialized: now 1790430317705971 us; error 26 us; skew 500 ppm
Sep 26 13:45:17 dist-test-slave-hc1k krb5kdc[18510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430317, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:17.708493 18751 init.cc:377] Logged in from keytab as kudu/127.12.170.66@KRBTEST.COM (short username kudu)
I20260926 13:45:17.709264 18751 webserver.cc:460] Webserver started at http://127.12.170.66:39047/ using document root <none> and password file <none>
I20260926 13:45:17.709749 18751 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:17.709808 18751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:17.709975 18751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:17.711947 18751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-1/data/instance:
uuid: "514934b7e0c141d381abde91650c23a2"
format_stamp: "Formatted at 2026-09-26 13:45:17 on dist-test-slave-hc1k"
server_key: "ba0a84d9b04fb783b3c87aec1e470cce"
server_key_iv: "646b63f95564b5baa1a278650b8d7adf"
server_key_version: "encryptionkey@0"
I20260926 13:45:17.712466 18751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance:
uuid: "514934b7e0c141d381abde91650c23a2"
format_stamp: "Formatted at 2026-09-26 13:45:17 on dist-test-slave-hc1k"
server_key: "ba0a84d9b04fb783b3c87aec1e470cce"
server_key_iv: "646b63f95564b5baa1a278650b8d7adf"
server_key_version: "encryptionkey@0"
I20260926 13:45:17.715873 18751 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:45:17.718134 18766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:17.719359 18751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:17.719480 18751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "514934b7e0c141d381abde91650c23a2"
format_stamp: "Formatted at 2026-09-26 13:45:17 on dist-test-slave-hc1k"
server_key: "ba0a84d9b04fb783b3c87aec1e470cce"
server_key_iv: "646b63f95564b5baa1a278650b8d7adf"
server_key_version: "encryptionkey@0"
I20260926 13:45:17.719566 18751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:17.737910 18751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:17.740728 18751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:17.740882 18751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:17.741392 18751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:17.742151 18751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:17.742195 18751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:17.742244 18751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:17.742282 18751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:17.750025 18751 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:45917
I20260926 13:45:17.750118 18879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:45917 every 8 connection(s)
I20260926 13:45:17.750831 18751 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:45:17.758677 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 18751
I20260926 13:45:17.758782 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance
I20260926 13:45:17.759034 12969 external_mini_cluster.cc:1428] Setting key 9020aef39a659da999e250c6346d26e4
Sep 26 13:45:17 dist-test-slave-hc1k krb5kdc[18510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430317, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:17.763713 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:17.752469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:55631 (local address 127.12.170.126:34021)
0926 13:45:17.752596 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:17.752599 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:17.753346 (+ 747us) server_negotiation.cc:407] Connection header received
0926 13:45:17.754645 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:17.754650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:17.754712 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:17.754794 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:17.755995 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:17.756528 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:17.757664 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:17.757899 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:17.757992 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:17.760476 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:17.760479 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:17.760481 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:17.760503 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:17.761766 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:17.762236 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:17.762240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:17.762242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:17.762339 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:17.762615 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:17.762619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:17.762621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:17.762902 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:17.763001 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:17.763405 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 13:45:17.763531 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 13:45:17.764477 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:17.752777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34021 (local address 127.12.170.66:55631)
0926 13:45:17.753198 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:17.753239 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:17.754439 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:17.754991 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:17.754998 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:17.755347 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:17.755848 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:17.755865 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:17.756654 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:17.756657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:17.757538 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:17.757548 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:17.758047 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:17.758052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:17.758216 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:17.758736 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:17.758752 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:17.760326 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:17.761894 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:17.761899 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:17.761912 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:17.762152 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:17.762420 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:17.762423 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:17.762425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:17.762537 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:17.762991 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:17.762995 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:17.763187 (+ 192us) client_negotiation.cc:769] Sending connection context
0926 13:45:17.763338 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 13:45:17.763620 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":119,"threads_started":1}
I20260926 13:45:17.765460 18880 heartbeater.cc:347] Connected to a master server at 127.12.170.126:34021
I20260926 13:45:17.765698 18880 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:17.766232 18880 heartbeater.cc:511] Master 127.12.170.126:34021 requested a full tablet report, sending...
I20260926 13:45:17.767254 18555 ts_manager.cc:194] Registered new tserver with Master: 514934b7e0c141d381abde91650c23a2 (127.12.170.66:45917)
I20260926 13:45:17.767827 18555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.66@KRBTEST.COM'} at 127.12.170.66:55631
WARNING: no policy specified for kudu/127.12.170.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
I20260926 13:45:17.784078 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:0
--local_ip_for_outbound_sockets=127.12.170.67
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:34021
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:40361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:17.887941 18885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:17.888156 18885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:17.888199 18885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:17.892071 18885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:17.892117 18885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:17.892170 18885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:45:17.897261 18885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:40361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:34021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:17.898126 18885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:17.898999 18885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:17.907629 18891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:17.907794 18885 server_base.cc:1034] running on GCE node
W20260926 13:45:17.907636 18893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:17.907944 18890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:17.908314 18885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:17.908883 18885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:17.910036 18885 hybrid_clock.cc:648] HybridClock initialized: now 1790430317910021 us; error 26 us; skew 500 ppm
Sep 26 13:45:17 dist-test-slave-hc1k krb5kdc[18510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430317, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:17.919495 18885 init.cc:377] Logged in from keytab as kudu/127.12.170.67@KRBTEST.COM (short username kudu)
I20260926 13:45:17.920358 18885 webserver.cc:460] Webserver started at http://127.12.170.67:42745/ using document root <none> and password file <none>
I20260926 13:45:17.920862 18885 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:17.920923 18885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:17.921094 18885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:17.922986 18885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-2/data/instance:
uuid: "6b527bcf11ac49d7862c3da168c160ee"
format_stamp: "Formatted at 2026-09-26 13:45:17 on dist-test-slave-hc1k"
server_key: "3b87432049117b99deb0b2ac25797f3f"
server_key_iv: "fe7ff570b7b8f59700485632aa71633c"
server_key_version: "encryptionkey@0"
I20260926 13:45:17.923506 18885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance:
uuid: "6b527bcf11ac49d7862c3da168c160ee"
format_stamp: "Formatted at 2026-09-26 13:45:17 on dist-test-slave-hc1k"
server_key: "3b87432049117b99deb0b2ac25797f3f"
server_key_iv: "fe7ff570b7b8f59700485632aa71633c"
server_key_version: "encryptionkey@0"
I20260926 13:45:17.926844 18885 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 13:45:17.929029 18900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:17.930099 18885 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:17.930202 18885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "6b527bcf11ac49d7862c3da168c160ee"
format_stamp: "Formatted at 2026-09-26 13:45:17 on dist-test-slave-hc1k"
server_key: "3b87432049117b99deb0b2ac25797f3f"
server_key_iv: "fe7ff570b7b8f59700485632aa71633c"
server_key_version: "encryptionkey@0"
I20260926 13:45:17.930281 18885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:17.947450 18885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:17.949834 18885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:17.949981 18885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:17.950456 18885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:17.951144 18885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:17.951186 18885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:17.951232 18885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:17.951258 18885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:17.958918 18885 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:37909
I20260926 13:45:17.958952 19013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:37909 every 8 connection(s)
I20260926 13:45:17.959697 18885 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
Sep 26 13:45:17 dist-test-slave-hc1k krb5kdc[18510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430317, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:17.968922 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 18885
I20260926 13:45:17.969019 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance
I20260926 13:45:17.969302 12969 external_mini_cluster.cc:1428] Setting key 11ad690a633b51b3f49a98860f535515
I20260926 13:45:17.971874 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:17.961285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:51711 (local address 127.12.170.126:34021)
0926 13:45:17.961432 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:17.961436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:17.962113 (+ 677us) server_negotiation.cc:407] Connection header received
0926 13:45:17.963338 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:17.963341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:17.963394 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:17.963465 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:17.964531 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:17.965045 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:17.966107 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:17.966320 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:17.966406 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:17.968539 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:17.968542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:17.968544 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:17.968565 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:17.969977 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:17.970436 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:17.970440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:17.970442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:17.970548 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:17.970872 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:17.970876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:17.970878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:17.971085 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:17.971178 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:17.971637 (+ 459us) server_negotiation.cc:299] Negotiation successful
0926 13:45:17.971760 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 13:45:17.972537 19016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:17.961596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34021 (local address 127.12.170.67:51711)
0926 13:45:17.961983 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:17.962020 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:17.963150 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:17.963624 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:17.963630 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:17.963978 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:17.964400 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:17.964412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:17.965151 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:17.965154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:17.965992 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:17.966001 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:17.966450 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:17.966454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:17.966599 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:17.967059 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:17.967075 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:17.968396 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:17.970062 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:17.970067 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:17.970079 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:17.970337 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:17.970633 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:17.970636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:17.970638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:17.970784 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:17.971168 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:17.971174 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:17.971427 (+ 253us) client_negotiation.cc:769] Sending connection context
0926 13:45:17.971586 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 13:45:17.971843 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":107,"threads_started":1}
I20260926 13:45:17.973425 19014 heartbeater.cc:347] Connected to a master server at 127.12.170.126:34021
I20260926 13:45:17.973618 19014 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:17.974006 19014 heartbeater.cc:511] Master 127.12.170.126:34021 requested a full tablet report, sending...
I20260926 13:45:17.974907 18555 ts_manager.cc:194] Registered new tserver with Master: 6b527bcf11ac49d7862c3da168c160ee (127.12.170.67:37909)
I20260926 13:45:17.975423 18555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.67@KRBTEST.COM'} at 127.12.170.67:51711
I20260926 13:45:17.982797 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/test-admin.keytab.
Sep 26 13:45:17 dist-test-slave-hc1k krb5kdc[18510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:17.996829 12969 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 13:45:17.999094 19021 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:17.997995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40394 (local address 127.12.170.65:46487)
0926 13:45:17.998290 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:17.998300 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:17.998321 (+ 21us) server_negotiation.cc:407] Connection header received
0926 13:45:17.998365 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:17.998369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:17.998537 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:17.998642 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:17.998968 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40394: BlockingRecv error: recv got EOF from 127.0.0.1:40394 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":116,"threads_started":1}
W20260926 13:45:17.999229 12969 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/krb5cc' of type FILE
Sep 26 13:45:17 dist-test-slave-hc1k krb5kdc[18510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:18.000142 12969 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 13:45:17 dist-test-slave-hc1k krb5kdc[18510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:18.000950 12969 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 13:45:18 dist-test-slave-hc1k krb5kdc[18510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.65@KRBTEST.COM
I20260926 13:45:18.009331 19021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:18.001683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40402 (local address 127.12.170.65:46487)
0926 13:45:18.001790 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:18.001794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:18.001848 (+ 54us) server_negotiation.cc:407] Connection header received
0926 13:45:18.001970 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:18.001973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:18.002015 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:18.002072 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:18.002544 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:18.003279 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:18.004097 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:18.004453 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:18.004542 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:18.006493 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:18.006502 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:18.006513 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:18.006533 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:18.007797 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:18.008162 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:18.008168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:18.008170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:18.008246 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:18.008479 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:18.008482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:18.008483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:18.008693 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:18.008892 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:18.009061 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 13:45:18.009197 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 13:45:18.012692 19021 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:18.011855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40404 (local address 127.12.170.65:46487)
0926 13:45:18.011973 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:18.011977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:18.012084 (+ 107us) server_negotiation.cc:407] Connection header received
0926 13:45:18.012204 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:18.012206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:18.012238 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:18.012320 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:18.012614 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40404: BlockingRecv error: recv got EOF from 127.0.0.1:40404 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 13:45:18.012853 12969 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/krb5cc' of type FILE
Sep 26 13:45:18 dist-test-slave-hc1k krb5kdc[18510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:18.013746 12969 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 13:45:18 dist-test-slave-hc1k krb5kdc[18510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:18.014513 12969 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 13:45:18 dist-test-slave-hc1k krb5kdc[18510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.65@KRBTEST.COM
I20260926 13:45:18.021884 19021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:18.015168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40416 (local address 127.12.170.65:46487)
0926 13:45:18.015275 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:18.015278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:18.015343 (+ 65us) server_negotiation.cc:407] Connection header received
0926 13:45:18.015449 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:18.015452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:18.015485 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:18.015545 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:18.016043 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:18.016538 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:18.017249 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:18.017467 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:18.017545 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:18.019348 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:18.019351 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:18.019354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:18.019374 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:18.020568 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:18.020925 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:18.020928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:18.020929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:18.020991 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:18.021213 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:18.021216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:18.021218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:18.021351 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:18.021451 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:18.021668 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 13:45:18.021759 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 13:45:18.024041 19021 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:18.023231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40426 (local address 127.12.170.65:46487)
0926 13:45:18.023373 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:18.023376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:18.023437 (+ 61us) server_negotiation.cc:407] Connection header received
0926 13:45:18.023548 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:18.023550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:18.023582 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:18.023640 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:18.023966 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40426: BlockingRecv error: recv got EOF from 127.0.0.1:40426 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 13:45:18.024130 12969 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790430291939677-12969-0/krb5kdc/krb5cc' of type FILE
Sep 26 13:45:18 dist-test-slave-hc1k krb5kdc[18510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:18.024904 12969 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 13:45:18 dist-test-slave-hc1k krb5kdc[18510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:18.025629 12969 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 13:45:18 dist-test-slave-hc1k krb5kdc[18510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.65@KRBTEST.COM
I20260926 13:45:18.032656 19021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:18.026265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40434 (local address 127.12.170.65:46487)
0926 13:45:18.026363 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:18.026367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:18.026429 (+ 62us) server_negotiation.cc:407] Connection header received
0926 13:45:18.026535 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:18.026537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:18.026572 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:18.026627 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:18.027041 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:18.027491 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:18.028228 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:18.028442 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:18.028520 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:18.030323 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:18.030326 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:18.030328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:18.030349 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:18.031328 (+ 979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:18.031690 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:18.031693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:18.031695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:18.031768 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:18.031988 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:18.031991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:18.031993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:18.032130 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:18.032228 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:18.032440 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 13:45:18.032527 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 13:45:18.033463 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 18617
I20260926 13:45:18.038681 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 18751
I20260926 13:45:18.042927 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 18885
I20260926 13:45:18.047330 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 18523
2026-09-26T13:45:18Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1978 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:45:18 dist-test-slave-hc1k krb5kdc[19034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:45:18 dist-test-slave-hc1k krb5kdc[19034](info): set up 2 sockets
Sep 26 13:45:18 dist-test-slave-hc1k krb5kdc[19034](info): commencing operation
krb5kdc: starting...
W20260926 13:45:18.987548 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:45:19 dist-test-slave-hc1k krb5kdc[19034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:45:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:45:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.126.keytab.
Entry for principal oryx/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.126.keytab.
I20260926 13:45:19.096789 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:33065
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.126.keytab
--principal=oryx/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:42035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:33065
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:19.198896 19047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:19.199126 19047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:19.199162 19047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:19.202989 19047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:45:19.203033 19047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:19.203049 19047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:19.203064 19047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:45:19.203078 19047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:45:19.207978 19047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:42035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:33065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.126.keytab
--principal=oryx/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:33065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:19.208818 19047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:19.209685 19047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:19.217423 19055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:19.217399 19053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:19.217398 19052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:19.217836 19047 server_base.cc:1034] running on GCE node
I20260926 13:45:19.218323 19047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:19.219277 19047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:19.220427 19047 hybrid_clock.cc:648] HybridClock initialized: now 1790430319220413 us; error 28 us; skew 500 ppm
Sep 26 13:45:19 dist-test-slave-hc1k krb5kdc[19034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430319, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:19.222893 19047 init.cc:377] Logged in from keytab as oryx/127.12.170.126@KRBTEST.COM (short username oryx)
I20260926 13:45:19.223716 19047 webserver.cc:460] Webserver started at http://127.12.170.126:35681/ using document root <none> and password file <none>
I20260926 13:45:19.224190 19047 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:19.224242 19047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:19.224397 19047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:19.226222 19047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "0f4afcedc19e4050b7b7320081502c13"
format_stamp: "Formatted at 2026-09-26 13:45:19 on dist-test-slave-hc1k"
server_key: "05da580eff368972b1f2dd658e76c5b2"
server_key_iv: "5524d08b78a88170b4c9272621275361"
server_key_version: "encryptionkey@0"
I20260926 13:45:19.226691 19047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "0f4afcedc19e4050b7b7320081502c13"
format_stamp: "Formatted at 2026-09-26 13:45:19 on dist-test-slave-hc1k"
server_key: "05da580eff368972b1f2dd658e76c5b2"
server_key_iv: "5524d08b78a88170b4c9272621275361"
server_key_version: "encryptionkey@0"
I20260926 13:45:19.229976 19047 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:45:19.232062 19062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:19.233147 19047 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:19.233256 19047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "0f4afcedc19e4050b7b7320081502c13"
format_stamp: "Formatted at 2026-09-26 13:45:19 on dist-test-slave-hc1k"
server_key: "05da580eff368972b1f2dd658e76c5b2"
server_key_iv: "5524d08b78a88170b4c9272621275361"
server_key_version: "encryptionkey@0"
I20260926 13:45:19.233338 19047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:19.249017 19047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:19.251717 19047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:19.251886 19047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:19.258118 19047 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:33065
I20260926 13:45:19.258347 19114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:33065 every 8 connection(s)
I20260926 13:45:19.259001 19047 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:45:19.261317 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 19047
I20260926 13:45:19.261421 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:45:19.261560 19115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:19.261699 12969 external_mini_cluster.cc:1428] Setting key 2ff07224d51ca3589bd8f74fa45cef98
I20260926 13:45:19.268131 19115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13: Bootstrap starting.
Sep 26 13:45:19 dist-test-slave-hc1k krb5kdc[19034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM
I20260926 13:45:19.270311 19115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:19.271036 19115 log.cc:824] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:19.272933 19115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13: No bootstrap required, opened a new log
I20260926 13:45:19.275161 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:19.262949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48546 (local address 127.12.170.126:33065)
0926 13:45:19.263322 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:19.263340 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:19.263380 (+ 40us) server_negotiation.cc:407] Connection header received
0926 13:45:19.264046 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:19.264068 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:19.264466 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:19.264816 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:19.265417 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:19.266873 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:19.267730 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:19.268379 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:19.268529 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:19.270827 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:19.270834 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:19.270850 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:19.270878 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:19.272484 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:19.273055 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:19.273061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:19.273066 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:19.273178 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:19.273478 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:19.273480 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:19.273481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:19.273818 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:19.273975 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:19.274255 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 13:45:19.274488 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":96,"threads_started":1}
I20260926 13:45:19.276774 19115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } }
I20260926 13:45:19.276995 19115 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:19.277082 19115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f4afcedc19e4050b7b7320081502c13, State: Initialized, Role: FOLLOWER
I20260926 13:45:19.277489 19115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } }
I20260926 13:45:19.277619 19115 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:45:19.277686 19115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:45:19.277784 19115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:19.278614 19115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } }
I20260926 13:45:19.278920 19115 leader_election.cc:302] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f4afcedc19e4050b7b7320081502c13; no voters:
I20260926 13:45:19.279196 19115 leader_election.cc:288] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:45:19.279287 19120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:19.279488 19120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [term 1 LEADER]: Becoming Leader. State: Replica: 0f4afcedc19e4050b7b7320081502c13, State: Running, Role: LEADER
I20260926 13:45:19.279834 19120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } }
I20260926 13:45:19.280323 19115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:45:19.281309 19122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f4afcedc19e4050b7b7320081502c13. Latest consensus state: current_term: 1 leader_uuid: "0f4afcedc19e4050b7b7320081502c13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } }
I20260926 13:45:19.281427 19122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:19.281620 19121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f4afcedc19e4050b7b7320081502c13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } }
I20260926 13:45:19.281755 19121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:19.281814 19129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:45:19.284791 19129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:45:19.290056 19129 catalog_manager.cc:1347] Generated new cluster ID: d34e7363d79146f9bc88ca2ccf28bda6
I20260926 13:45:19.290117 19129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:45:19.306270 19129 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:45:19.307412 19129 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:45:19.317181 19129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13: Generated new TSK 0
I20260926 13:45:19.317859 19129 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.170.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.170.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.170.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.65.keytab.
Entry for principal oryx/127.12.170.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.65.keytab.
I20260926 13:45:19.342341 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:0
--local_ip_for_outbound_sockets=127.12.170.65
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:33065
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.65.keytab
--principal=oryx/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:42035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:19.445734 19141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:19.445959 19141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:19.445996 19141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:19.449867 19141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:19.449916 19141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:19.449975 19141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:45:19.454784 19141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:42035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.65.keytab
--principal=oryx/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:33065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:19.455626 19141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:19.456554 19141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:19.465283 19146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:19.465314 19149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:19.465379 19141 server_base.cc:1034] running on GCE node
W20260926 13:45:19.465312 19147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:19.465830 19141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:19.466398 19141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:19.467556 19141 hybrid_clock.cc:648] HybridClock initialized: now 1790430319467541 us; error 26 us; skew 500 ppm
Sep 26 13:45:19 dist-test-slave-hc1k krb5kdc[19034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430319, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:19.469966 19141 init.cc:377] Logged in from keytab as oryx/127.12.170.65@KRBTEST.COM (short username oryx)
I20260926 13:45:19.470695 19141 webserver.cc:460] Webserver started at http://127.12.170.65:40739/ using document root <none> and password file <none>
I20260926 13:45:19.471175 19141 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:19.471235 19141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:19.471400 19141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:19.473255 19141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/data/instance:
uuid: "144cccc67d584072b304beeb25acad33"
format_stamp: "Formatted at 2026-09-26 13:45:19 on dist-test-slave-hc1k"
server_key: "7ebf4d9612cd05608fafb5dda261e91f"
server_key_iv: "6729515311ab76e4243329716e1a4d78"
server_key_version: "encryptionkey@0"
I20260926 13:45:19.473748 19141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance:
uuid: "144cccc67d584072b304beeb25acad33"
format_stamp: "Formatted at 2026-09-26 13:45:19 on dist-test-slave-hc1k"
server_key: "7ebf4d9612cd05608fafb5dda261e91f"
server_key_iv: "6729515311ab76e4243329716e1a4d78"
server_key_version: "encryptionkey@0"
I20260926 13:45:19.477029 19141 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:45:19.479066 19156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:19.480183 19141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:45:19.480278 19141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "144cccc67d584072b304beeb25acad33"
format_stamp: "Formatted at 2026-09-26 13:45:19 on dist-test-slave-hc1k"
server_key: "7ebf4d9612cd05608fafb5dda261e91f"
server_key_iv: "6729515311ab76e4243329716e1a4d78"
server_key_version: "encryptionkey@0"
I20260926 13:45:19.480360 19141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:19.496273 19141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:19.498750 19141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:19.498904 19141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:19.499408 19141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:19.500181 19141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:19.500223 19141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:19.500264 19141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:19.500293 19141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:19.508237 19141 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:33411
I20260926 13:45:19.508288 19269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:33411 every 8 connection(s)
I20260926 13:45:19.509034 19141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:45:19.517374 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 19141
I20260926 13:45:19.517488 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance
Sep 26 13:45:19 dist-test-slave-hc1k krb5kdc[19034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430319, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM
I20260926 13:45:19.517758 12969 external_mini_cluster.cc:1428] Setting key 549567bc38e72f4aa5859ff7884bc335
I20260926 13:45:19.521673 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:19.510625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:55555 (local address 127.12.170.126:33065)
0926 13:45:19.510750 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:19.510755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:19.511578 (+ 823us) server_negotiation.cc:407] Connection header received
0926 13:45:19.512842 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:19.512845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:19.512890 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:19.512965 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:19.514096 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:19.514614 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:19.515643 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:19.515871 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:19.515949 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:19.518228 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:19.518232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:19.518234 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:19.518257 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:19.519733 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:19.520264 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:19.520268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:19.520270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:19.520358 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:19.520654 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:19.520657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:19.520659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:19.520836 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:19.520920 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:19.521383 (+ 463us) server_negotiation.cc:299] Negotiation successful
0926 13:45:19.521500 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:45:19.522434 19272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:19.511010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.12.170.65:55555)
0926 13:45:19.511444 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:19.511484 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:19.512664 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:19.513103 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:19.513110 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:19.513445 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:19.513970 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:19.513987 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:19.514702 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:19.514704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:19.515558 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:19.515567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:19.515992 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:19.515995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:19.516137 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:19.516652 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:19.516667 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:19.518075 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:19.519862 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:19.519868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:19.519882 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:19.520173 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:19.520437 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:19.520440 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:19.520442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:19.520568 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:19.520931 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:19.520938 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:19.521159 (+ 221us) client_negotiation.cc:769] Sending connection context
0926 13:45:19.521315 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 13:45:19.521630 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":109,"threads_started":1}
I20260926 13:45:19.523540 19270 heartbeater.cc:347] Connected to a master server at 127.12.170.126:33065
I20260926 13:45:19.523777 19270 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:19.524212 19270 heartbeater.cc:511] Master 127.12.170.126:33065 requested a full tablet report, sending...
I20260926 13:45:19.525713 19079 ts_manager.cc:194] Registered new tserver with Master: 144cccc67d584072b304beeb25acad33 (127.12.170.65:33411)
WARNING: no policy specified for oryx/127.12.170.66@KRBTEST.COM; defaulting to no policy
I20260926 13:45:19.526911 19079 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.170.65@KRBTEST.COM'} at 127.12.170.65:55555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.170.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.170.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.66.keytab.
Entry for principal oryx/127.12.170.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.66.keytab.
I20260926 13:45:19.540000 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:0
--local_ip_for_outbound_sockets=127.12.170.66
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:33065
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.66.keytab
--principal=oryx/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:42035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:19.641183 19275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:19.641392 19275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:19.641429 19275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:19.645332 19275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:19.645378 19275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:19.645440 19275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:45:19.650321 19275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:42035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.66.keytab
--principal=oryx/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:33065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:19.651167 19275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:19.652061 19275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:19.660846 19283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:19.660904 19280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:19.661051 19275 server_base.cc:1034] running on GCE node
W20260926 13:45:19.660884 19281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:19.661437 19275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:19.662014 19275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:19.663167 19275 hybrid_clock.cc:648] HybridClock initialized: now 1790430319663144 us; error 38 us; skew 500 ppm
Sep 26 13:45:19 dist-test-slave-hc1k krb5kdc[19034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430319, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:19.665622 19275 init.cc:377] Logged in from keytab as oryx/127.12.170.66@KRBTEST.COM (short username oryx)
I20260926 13:45:19.666335 19275 webserver.cc:460] Webserver started at http://127.12.170.66:40589/ using document root <none> and password file <none>
I20260926 13:45:19.666787 19275 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:19.666838 19275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:19.666998 19275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:19.668789 19275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/data/instance:
uuid: "5378a400ff1e490fb8c805c8465a4f37"
format_stamp: "Formatted at 2026-09-26 13:45:19 on dist-test-slave-hc1k"
server_key: "01c531701f41a8036003ace7ebbc93a6"
server_key_iv: "7c8ac42d47ca893b5a659dad63ea4fe0"
server_key_version: "encryptionkey@0"
I20260926 13:45:19.669246 19275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance:
uuid: "5378a400ff1e490fb8c805c8465a4f37"
format_stamp: "Formatted at 2026-09-26 13:45:19 on dist-test-slave-hc1k"
server_key: "01c531701f41a8036003ace7ebbc93a6"
server_key_iv: "7c8ac42d47ca893b5a659dad63ea4fe0"
server_key_version: "encryptionkey@0"
I20260926 13:45:19.672520 19275 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:45:19.674597 19290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:19.675704 19275 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:45:19.675789 19275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "5378a400ff1e490fb8c805c8465a4f37"
format_stamp: "Formatted at 2026-09-26 13:45:19 on dist-test-slave-hc1k"
server_key: "01c531701f41a8036003ace7ebbc93a6"
server_key_iv: "7c8ac42d47ca893b5a659dad63ea4fe0"
server_key_version: "encryptionkey@0"
I20260926 13:45:19.675863 19275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:19.688732 19275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:19.691368 19275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:19.691496 19275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:19.691991 19275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:19.692662 19275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:19.692703 19275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:19.692737 19275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:19.692756 19275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:19.700349 19275 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:33105
I20260926 13:45:19.700380 19403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:33105 every 8 connection(s)
I20260926 13:45:19.701092 19275 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:45:19.704677 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 19275
I20260926 13:45:19.704797 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance
I20260926 13:45:19.705096 12969 external_mini_cluster.cc:1428] Setting key 2bef1b5a356b82294a2986cdc196b98c
Sep 26 13:45:19 dist-test-slave-hc1k krb5kdc[19034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430319, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.12.170.67@KRBTEST.COM; defaulting to no policy
I20260926 13:45:19.713883 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:19.702600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:43589 (local address 127.12.170.126:33065)
0926 13:45:19.702719 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:19.702722 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:19.703442 (+ 720us) server_negotiation.cc:407] Connection header received
0926 13:45:19.704791 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:19.704795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:19.704850 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:19.704931 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:19.706160 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:19.706760 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:19.708279 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:19.708517 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:19.708607 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:19.710756 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:19.710759 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:19.710762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:19.710787 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:19.712076 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:19.712519 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:19.712522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:19.712524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:19.712597 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:19.712889 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:19.712892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:19.712894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:19.713059 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:19.713142 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:19.713604 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 13:45:19.713725 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:45:19.714628 19406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:19.702898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.12.170.66:43589)
0926 13:45:19.703306 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:19.703347 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:19.704552 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:19.705097 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:19.705105 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:19.705525 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:19.706037 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:19.706050 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:19.706884 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:19.706888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:19.708170 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:19.708180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:19.708647 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:19.708651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:19.708808 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:19.709308 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:19.709324 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:19.710608 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:19.712175 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:19.712180 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:19.712193 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:19.712424 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:19.712674 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:19.712677 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:19.712679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:19.712800 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:19.713150 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:19.713156 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:19.713384 (+ 228us) client_negotiation.cc:769] Sending connection context
0926 13:45:19.713547 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 13:45:19.713843 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.170.67@KRBTEST.COM" created.
I20260926 13:45:19.715626 19404 heartbeater.cc:347] Connected to a master server at 127.12.170.126:33065
I20260926 13:45:19.715878 19404 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:19.716382 19404 heartbeater.cc:511] Master 127.12.170.126:33065 requested a full tablet report, sending...
I20260926 13:45:19.717373 19079 ts_manager.cc:194] Registered new tserver with Master: 5378a400ff1e490fb8c805c8465a4f37 (127.12.170.66:33105)
I20260926 13:45:19.718045 19079 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.170.66@KRBTEST.COM'} at 127.12.170.66:43589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.170.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.67.keytab.
Entry for principal oryx/127.12.170.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.67.keytab.
I20260926 13:45:19.727895 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:0
--local_ip_for_outbound_sockets=127.12.170.67
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:33065
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.67.keytab
--principal=oryx/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:42035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:19.830461 19409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:19.830691 19409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:19.830731 19409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:19.834645 19409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:19.834695 19409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:19.834749 19409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:45:19.839813 19409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:42035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.67.keytab
--principal=oryx/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:33065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:19.840726 19409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:19.841689 19409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:19.850577 19417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:19.850602 19415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:19.850602 19414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:19.850646 19409 server_base.cc:1034] running on GCE node
I20260926 13:45:19.851177 19409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:19.851788 19409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:19.852975 19409 hybrid_clock.cc:648] HybridClock initialized: now 1790430319852948 us; error 41 us; skew 500 ppm
Sep 26 13:45:19 dist-test-slave-hc1k krb5kdc[19034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430319, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:19.855459 19409 init.cc:377] Logged in from keytab as oryx/127.12.170.67@KRBTEST.COM (short username oryx)
I20260926 13:45:19.856221 19409 webserver.cc:460] Webserver started at http://127.12.170.67:43189/ using document root <none> and password file <none>
I20260926 13:45:19.856688 19409 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:19.856740 19409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:19.856891 19409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:19.858842 19409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/data/instance:
uuid: "875eb6152ee6435683b49d64a8986403"
format_stamp: "Formatted at 2026-09-26 13:45:19 on dist-test-slave-hc1k"
server_key: "8f80528cb9c6e86d44674edb91c6fca1"
server_key_iv: "24cece39732052991783e12b77f0e68e"
server_key_version: "encryptionkey@0"
I20260926 13:45:19.859316 19409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance:
uuid: "875eb6152ee6435683b49d64a8986403"
format_stamp: "Formatted at 2026-09-26 13:45:19 on dist-test-slave-hc1k"
server_key: "8f80528cb9c6e86d44674edb91c6fca1"
server_key_iv: "24cece39732052991783e12b77f0e68e"
server_key_version: "encryptionkey@0"
I20260926 13:45:19.862746 19409 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 13:45:19.864977 19424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:19.866129 19409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:19.866219 19409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "875eb6152ee6435683b49d64a8986403"
format_stamp: "Formatted at 2026-09-26 13:45:19 on dist-test-slave-hc1k"
server_key: "8f80528cb9c6e86d44674edb91c6fca1"
server_key_iv: "24cece39732052991783e12b77f0e68e"
server_key_version: "encryptionkey@0"
I20260926 13:45:19.866312 19409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:19.893457 19409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:19.896116 19409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:19.896270 19409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:19.896808 19409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:19.897554 19409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:19.897598 19409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:19.897639 19409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:19.897667 19409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:19.905941 19409 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:34977
I20260926 13:45:19.905973 19537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:34977 every 8 connection(s)
I20260926 13:45:19.906742 19409 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:45:19.912974 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 19409
I20260926 13:45:19.913074 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance
I20260926 13:45:19.913321 12969 external_mini_cluster.cc:1428] Setting key a5aa78a693ecc2476e4d64f1bbecd68b
Sep 26 13:45:19 dist-test-slave-hc1k krb5kdc[19034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430319, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM
I20260926 13:45:19.919186 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:19.908359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:48679 (local address 127.12.170.126:33065)
0926 13:45:19.908479 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:19.908483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:19.909263 (+ 780us) server_negotiation.cc:407] Connection header received
0926 13:45:19.910538 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:19.910541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:19.910599 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:19.910668 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:19.911819 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:19.912339 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:19.913507 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:19.913770 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:19.913855 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:19.916406 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:19.916409 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:19.916411 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:19.916437 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:19.917676 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:19.918072 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:19.918074 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:19.918075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:19.918140 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:19.918372 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:19.918374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:19.918375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:19.918526 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:19.918591 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:19.918988 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 13:45:19.919076 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 13:45:19.919988 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:19.908677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.12.170.67:48679)
0926 13:45:19.909118 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:19.909158 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:19.910332 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:19.910844 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:19.910851 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:19.911198 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:19.911680 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:19.911700 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:19.912456 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:19.912459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:19.913364 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:19.913374 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:19.913929 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:19.913934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:19.914094 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:19.914731 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:19.914750 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:19.916274 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:19.917758 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:19.917762 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:19.917773 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:19.918004 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:19.918198 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:19.918200 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:19.918202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:19.918314 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:19.918592 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:19.918596 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:19.918797 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 13:45:19.918959 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 13:45:19.919190 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":109,"threads_started":1}
I20260926 13:45:19.920992 19538 heartbeater.cc:347] Connected to a master server at 127.12.170.126:33065
I20260926 13:45:19.921197 19538 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:19.921630 19538 heartbeater.cc:511] Master 127.12.170.126:33065 requested a full tablet report, sending...
I20260926 13:45:19.922520 19079 ts_manager.cc:194] Registered new tserver with Master: 875eb6152ee6435683b49d64a8986403 (127.12.170.67:34977)
I20260926 13:45:19.923072 19079 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.170.67@KRBTEST.COM'} at 127.12.170.67:48679
I20260926 13:45:19.926812 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.170.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.170.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.170.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.125.keytab.
Entry for principal oryx/127.12.170.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.125.keytab.
I20260926 13:45:19.950243 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.125:40913
--webserver_interface=127.12.170.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.125.keytab
--principal=oryx/127.12.170.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:42035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:33065,127.12.170.125:40913
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:20.053665 19543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:20.053929 19543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:20.053988 19543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:20.058041 19543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:45:20.058103 19543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:20.058128 19543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:20.058151 19543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:45:20.058177 19543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:45:20.063534 19543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:42035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:33065,127.12.170.125:40913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.125.keytab
--principal=oryx/127.12.170.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.125:40913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.170.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
W20260926 13:45:20.063771 19543 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.12.170.126:33065,127.12.170.125:40913'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 13:45:20.064630 19543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:20.065598 19543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:20.073436 19549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:20.073415 19548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:20.073429 19551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:20.073493 19543 server_base.cc:1034] running on GCE node
I20260926 13:45:20.074146 19543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:20.075121 19543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:20.076288 19543 hybrid_clock.cc:648] HybridClock initialized: now 1790430320076273 us; error 27 us; skew 500 ppm
Sep 26 13:45:20 dist-test-slave-hc1k krb5kdc[19034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430320, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:20.078887 19543 init.cc:377] Logged in from keytab as oryx/127.12.170.125@KRBTEST.COM (short username oryx)
I20260926 13:45:20.079780 19543 webserver.cc:460] Webserver started at http://127.12.170.125:39307/ using document root <none> and password file <none>
I20260926 13:45:20.080286 19543 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:20.080358 19543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:20.080562 19543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:20.082590 19543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-1/data/instance:
uuid: "8189bfdb296f41d69b7e564da513093f"
format_stamp: "Formatted at 2026-09-26 13:45:20 on dist-test-slave-hc1k"
server_key: "e71556e69353e1eeff69a2e76236d141"
server_key_iv: "2b3585a4c32e6aa25045d0d265380eb2"
server_key_version: "encryptionkey@0"
I20260926 13:45:20.083098 19543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-1/wal/instance:
uuid: "8189bfdb296f41d69b7e564da513093f"
format_stamp: "Formatted at 2026-09-26 13:45:20 on dist-test-slave-hc1k"
server_key: "e71556e69353e1eeff69a2e76236d141"
server_key_iv: "2b3585a4c32e6aa25045d0d265380eb2"
server_key_version: "encryptionkey@0"
I20260926 13:45:20.086643 19543 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260926 13:45:20.088884 19558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:20.090039 19543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:20.090147 19543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-1/wal
uuid: "8189bfdb296f41d69b7e564da513093f"
format_stamp: "Formatted at 2026-09-26 13:45:20 on dist-test-slave-hc1k"
server_key: "e71556e69353e1eeff69a2e76236d141"
server_key_iv: "2b3585a4c32e6aa25045d0d265380eb2"
server_key_version: "encryptionkey@0"
I20260926 13:45:20.090255 19543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:20.113754 19543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:20.116590 19543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:20.116765 19543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:20.123384 19543 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.125:40913
I20260926 13:45:20.123612 19610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.125:40913 every 8 connection(s)
I20260926 13:45:20.124347 19543 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-1/data/info.pb
I20260926 13:45:20.125299 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 19543
I20260926 13:45:20.127225 19611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:20.130726 19611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:45:20 dist-test-slave-hc1k krb5kdc[19034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.170.125@KRBTEST.COM
Sep 26 13:45:20 dist-test-slave-hc1k krb5kdc[19034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430320, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.125@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM
I20260926 13:45:20.138757 19614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.126672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60578 (local address 127.12.170.125:40913)
0926 13:45:20.127076 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:20.127090 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:20.127123 (+ 33us) server_negotiation.cc:407] Connection header received
0926 13:45:20.127785 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:20.127803 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:20.128126 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:20.128417 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:20.129159 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.130154 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.130967 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.131444 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.131593 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.134312 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:20.134318 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:20.134326 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:20.134358 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:20.136028 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.136521 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.136527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.136533 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.136676 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.136917 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.136920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.136922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.137341 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:20.137563 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:20.137793 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 13:45:20.138076 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":132,"threads_started":1}
I20260926 13:45:20.141213 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.132312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.0.0.1:48558)
0926 13:45:20.132677 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:20.132713 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:20.132823 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:20.133121 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:20.133127 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:20.133544 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:20.133718 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.133725 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.134518 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.134521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.135451 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.135462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.135967 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.135971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.136075 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.136521 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:20.136534 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:20.138117 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:20.139726 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.139732 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.139735 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.139953 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.140216 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.140218 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.140219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.140284 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.140743 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:20.140749 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:20.140844 (+ 95us) client_negotiation.cc:769] Sending connection context
0926 13:45:20.140957 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 13:45:20.141080 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":154,"threads_started":1}
I20260926 13:45:20.141219 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.132345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48558 (local address 127.12.170.126:33065)
0926 13:45:20.132459 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:20.132463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:20.132793 (+ 330us) server_negotiation.cc:407] Connection header received
0926 13:45:20.132942 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:20.132945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:20.132997 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:20.133061 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:20.133880 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.134400 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.135573 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.135862 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.135947 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.138243 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:20.138245 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:20.138247 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:20.138270 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:20.139616 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.140040 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.140043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.140045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.140135 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.140356 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.140358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.140360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.140637 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:20.140736 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:20.141014 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 13:45:20.141108 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:45:20.142366 19611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:45:20 dist-test-slave-hc1k krb5kdc[19034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430320, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.125@KRBTEST.COM for oryx/127.12.170.125@KRBTEST.COM
I20260926 13:45:20.150202 19614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.142850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60588 (local address 127.12.170.125:40913)
0926 13:45:20.142926 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:20.142930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:20.142942 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:45:20.143000 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:20.143002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:20.143034 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:20.143090 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:20.143412 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.144250 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.144925 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.145128 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.145205 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.147295 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:20.147298 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:20.147301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:20.147325 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:20.148888 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.149230 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.149234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.149235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.149305 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.149519 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.149522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.149524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.149672 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:20.149749 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:20.149971 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 13:45:20.150080 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":27}
I20260926 13:45:20.150202 19615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.142756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:40913 (local address 127.0.0.1:60588)
0926 13:45:20.142850 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:20.142864 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:20.142929 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:20.143108 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:20.143111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:20.143245 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:20.143316 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.143320 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.144342 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.144346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.144834 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.144840 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.145209 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.145212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.145288 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.145641 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:20.145651 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:20.146953 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:20.148982 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.148984 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.148987 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.149153 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.149375 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.149378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.149380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.149450 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.149748 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:20.149750 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:20.149827 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 13:45:20.149909 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 13:45:20.150029 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 13:45:20.152899 19611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f: Bootstrap starting.
I20260926 13:45:20.154978 19611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:20.155642 19611 log.cc:824] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:20.157357 19611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f: No bootstrap required, opened a new log
I20260926 13:45:20.160148 19611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } }
I20260926 13:45:20.160328 19611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:20.160390 19611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8189bfdb296f41d69b7e564da513093f, State: Initialized, Role: FOLLOWER
I20260926 13:45:20.160758 19611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } }
I20260926 13:45:20.161147 19617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } } }
I20260926 13:45:20.161252 19617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:45:20.161782 19611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [sys.catalog]: configured and running, proceeding with master startup.
W20260926 13:45:20.165942 19628 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:45:20.166004 19628 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:45:20.167766 19543 master_runner.cc:428] Detected that this master 8189bfdb296f41d69b7e564da513093f is joining an existing cluster
I20260926 13:45:20.167811 19543 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.170.125:40913
I20260926 13:45:20.177075 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.169677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:40913 (local address 127.0.0.1:60604)
0926 13:45:20.169921 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:20.169936 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:20.170046 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:20.170479 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:20.170481 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:20.170657 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:20.170781 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.170787 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.171445 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.171447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.172250 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.172258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.172807 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.172810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.172922 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.173369 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:20.173384 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:20.173784 (+ 400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:20.175506 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.175509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.175512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.175767 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.176050 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.176053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.176055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.176143 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.176530 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:20.176532 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:20.176602 (+ 70us) client_negotiation.cc:769] Sending connection context
0926 13:45:20.176709 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 13:45:20.176833 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":23,"spinlock_wait_cycles":14080,"thread_start_us":93,"threads_started":1}
I20260926 13:45:20.177176 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.169639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48568 (local address 127.12.170.126:33065)
0926 13:45:20.169733 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:20.169738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:20.170175 (+ 437us) server_negotiation.cc:407] Connection header received
0926 13:45:20.170262 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:20.170265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:20.170306 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:20.170373 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:20.170888 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.171702 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.172622 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.172881 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.172976 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.174154 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:20.174158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:20.174161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:20.174187 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:20.175624 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.176089 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.176093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.176095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.176169 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.176414 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.176417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.176418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.176576 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:20.176656 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:20.176944 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 13:45:20.177048 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 13:45:20.177086 19614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.169625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60604 (local address 127.12.170.125:40913)
0926 13:45:20.169721 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:20.169725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:20.169993 (+ 268us) server_negotiation.cc:407] Connection header received
0926 13:45:20.170133 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:20.170136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:20.170182 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:20.170260 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:20.170876 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.171354 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.172366 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.172697 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.172800 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.173896 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:20.173900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:20.173902 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:20.173925 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:20.175400 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.175883 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.175886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.175888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.175962 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.176234 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.176237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.176239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.176415 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:20.176479 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:20.176767 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 13:45:20.176907 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":38,"spinlock_wait_cycles":1792}
I20260926 13:45:20.177620 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.169750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.0.0.1:48568)
0926 13:45:20.170109 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:20.170122 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:20.170196 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:20.170398 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:20.170401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:20.170624 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:20.170758 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.170767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.171799 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.171803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.172529 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.172535 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.172990 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.172993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.173080 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.173518 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:20.173533 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:20.173999 (+ 466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:20.175745 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.175748 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.175751 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.175982 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.176246 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.176249 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.176251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.176337 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.176659 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:20.176662 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:20.176767 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 13:45:20.177349 (+ 582us) client_negotiation.cc:240] Negotiation successful
0926 13:45:20.177473 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"spinlock_wait_cycles":29184,"thread_start_us":105,"threads_started":1}
Sep 26 13:45:20 dist-test-slave-hc1k krb5kdc[19034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430319, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.126@KRBTEST.COM for oryx/127.12.170.125@KRBTEST.COM
I20260926 13:45:20.189399 19614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.181310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60608 (local address 127.12.170.125:40913)
0926 13:45:20.181405 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:20.181408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:20.181762 (+ 354us) server_negotiation.cc:407] Connection header received
0926 13:45:20.181892 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:20.181894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:20.181932 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:20.181991 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:20.182629 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.183142 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.184316 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.184512 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.184583 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.186695 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:20.186698 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:20.186700 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:20.186720 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:20.187938 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.188397 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.188399 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.188400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.188459 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.188678 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.188679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.188680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.188861 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:20.188931 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:20.189232 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 13:45:20.189305 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 13:45:20.189525 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.181279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:40913 (local address 127.0.0.1:60608)
0926 13:45:20.181615 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:20.181666 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:20.181782 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:20.182031 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:20.182036 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:20.182424 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:20.182539 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.182546 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.183233 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.183236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.184221 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.184231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.184601 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.184604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.184706 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.185120 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:20.185133 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:20.186562 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:20.188032 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.188041 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.188047 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.188317 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.188526 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.188528 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.188530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.188613 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.188944 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:20.188949 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:20.189063 (+ 114us) client_negotiation.cc:769] Sending connection context
0926 13:45:20.189197 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 13:45:20.189361 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":87,"threads_started":1}
I20260926 13:45:20.190867 19079 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.170.125:40913: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8189bfdb296f41d69b7e564da513093f"
member_type: NON_VOTER
last_known_addr {
host: "127.12.170.125"
port: 40913
}
attrs {
promote: true
}
}
dest_uuid: "0f4afcedc19e4050b7b7320081502c13"
cas_config_opid_index: -1
I20260926 13:45:20.191483 19079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: NON_VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: true } }
I20260926 13:45:20.194800 19637 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8189bfdb296f41d69b7e564da513093f (127.12.170.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: NON_VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: true } } }
I20260926 13:45:20.195421 19585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:20.197201 19585 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [term 1 FOLLOWER]: Refusing update from remote peer 0f4afcedc19e4050b7b7320081502c13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 13:45:20.197988 19639 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: NON_VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:20.199749 19637 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.170.125:40913
I20260926 13:45:20.200172 19543 master.cc:537] Master@127.12.170.125:40913 shutting down...
I20260926 13:45:20.200521 19637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0f4afcedc19e4050b7b7320081502c13" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: NON_VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: true } } }
I20260926 13:45:20.200683 19637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [sys.catalog]: This master's current role is: LEADER
WARNING: no policy specified for oryx/127.12.170.124@KRBTEST.COM; defaulting to no policy
I20260926 13:45:20.204867 19585 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8189bfdb296f41d69b7e564da513093f (127.12.170.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: NON_VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: true } } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.170.124@KRBTEST.COM" created.
I20260926 13:45:20.207111 19617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f4afcedc19e4050b7b7320081502c13. Latest consensus state: current_term: 1 leader_uuid: "0f4afcedc19e4050b7b7320081502c13" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: NON_VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: true } } }
I20260926 13:45:20.207134 19644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f4afcedc19e4050b7b7320081502c13" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: NON_VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: true } } }
I20260926 13:45:20.207209 19617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [sys.catalog]: This master's current role is: LEARNER
I20260926 13:45:20.207305 19617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f4afcedc19e4050b7b7320081502c13" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: NON_VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: true } } }
I20260926 13:45:20.207374 19617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [sys.catalog]: This master's current role is: LEARNER
I20260926 13:45:20.207448 19644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [sys.catalog]: This master's current role is: LEARNER
I20260926 13:45:20.208020 19642 mvcc.cc:205] Tried to move back new op lower bound from 7333602587920011264 to 7333602587771006976. Current Snapshot: MvccSnapshot[applied={T|T < 7333602587803504640}]
I20260926 13:45:20.208631 19543 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [term 1 LEARNER]: Raft consensus shutting down.
I20260926 13:45:20.208880 19543 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [term 1 LEARNER]: Raft consensus is shut down!
I20260926 13:45:20.208949 19543 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f: stopping tablet replica
I20260926 13:45:20.225240 19543 master.cc:552] Master@127.12.170.125:40913 shutdown complete.
I20260926 13:45:20.225292 19543 master_runner.cc:305] Clearing existing system tablet
I20260926 13:45:20.225515 19543 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:20.225569 19543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.170.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.124.keytab.
Entry for principal oryx/127.12.170.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.124.keytab.
I20260926 13:45:20.226234 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.124:39363
--webserver_interface=127.12.170.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.124.keytab
--principal=oryx/127.12.170.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:42035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:33065,127.12.170.125:40913,127.12.170.124:39363
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:45:20.226881 19543 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:45:20.227712 19652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:20.227898 19543 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:20.228031 19543 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:45:20.228072 19543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-1/wal
uuid: "8189bfdb296f41d69b7e564da513093f"
format_stamp: "Formatted at 2026-09-26 13:45:20 on dist-test-slave-hc1k"
server_key: "e71556e69353e1eeff69a2e76236d141"
server_key_iv: "2b3585a4c32e6aa25045d0d265380eb2"
server_key_version: "encryptionkey@0"
I20260926 13:45:20.229809 19543 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:20.232160 19543 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 13:45:20.232249 19543 log.cc:1196] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 13:45:20.232589 19543 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f: Deleting consensus metadata
I20260926 13:45:20.232746 19543 master_runner.cc:315] Copying system tablet from 127.12.170.126:33065
I20260926 13:45:20.233929 19543 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f: tablet copy: Beginning tablet copy session from remote peer at address 127.12.170.126:33065
I20260926 13:45:20.243738 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.234561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48584 (local address 127.12.170.126:33065)
0926 13:45:20.234687 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:20.234693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:20.235991 (+ 1298us) server_negotiation.cc:407] Connection header received
0926 13:45:20.236164 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:20.236168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:20.236223 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:20.236310 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:20.236820 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.237662 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.238810 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.239161 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.239259 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.240494 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:20.240498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:20.240501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:20.240529 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:20.242067 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.242544 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.242547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.242549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.242634 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.242899 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.242902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.242904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.243106 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:20.243199 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:20.243448 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 13:45:20.243557 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 13:45:20.243752 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.234448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.0.0.1:48584)
0926 13:45:20.235878 (+ 1430us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:20.235894 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:20.236023 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:20.236353 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:20.236356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:20.236560 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:20.236714 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.236722 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.237762 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.237765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.238694 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.238705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.239267 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.239271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.239372 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.239946 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:20.239960 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:20.240357 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:20.242180 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.242183 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.242186 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.242445 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.242719 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.242722 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.242724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.242815 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.243189 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:20.243192 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:20.243286 (+ 94us) client_negotiation.cc:769] Sending connection context
0926 13:45:20.243389 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 13:45:20.243520 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1356,"thread_start_us":77,"threads_started":1}
I20260926 13:45:20.244735 19099 tablet_copy_service.cc:140] P 0f4afcedc19e4050b7b7320081502c13: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8189bfdb296f41d69b7e564da513093f ({username='oryx', principal='oryx/127.12.170.125@KRBTEST.COM'} at 127.0.0.1:48584)
I20260926 13:45:20.244910 19099 tablet_copy_service.cc:161] P 0f4afcedc19e4050b7b7320081502c13: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8189bfdb296f41d69b7e564da513093f at {username='oryx', principal='oryx/127.12.170.125@KRBTEST.COM'} at 127.0.0.1:48584: session id = 8189bfdb296f41d69b7e564da513093f-00000000000000000000000000000000
I20260926 13:45:20.246541 19099 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 13:45:20.248119 19543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 13:45:20.250187 19543 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f: tablet copy: Starting download of 0 data blocks...
I20260926 13:45:20.250362 19543 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f: tablet copy: Starting download of 1 WAL segments...
I20260926 13:45:20.252027 19543 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 13:45:20.253585 19099 tablet_copy_service.cc:342] P 0f4afcedc19e4050b7b7320081502c13: Request end of tablet copy session 8189bfdb296f41d69b7e564da513093f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.170.125@KRBTEST.COM'} at 127.0.0.1:48584
I20260926 13:45:20.253691 19099 tablet_copy_service.cc:433] P 0f4afcedc19e4050b7b7320081502c13: ending tablet copy session 8189bfdb296f41d69b7e564da513093f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8189bfdb296f41d69b7e564da513093f
I20260926 13:45:20.263025 19543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:20.264652 19662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:20.265218 19660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:20.265486 19543 server_base.cc:1034] running on GCE node
W20260926 13:45:20.276048 19659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:20.276286 19543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:20.276582 19543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:20.277724 19543 hybrid_clock.cc:648] HybridClock initialized: now 1790430320277704 us; error 38 us; skew 500 ppm
Sep 26 13:45:20 dist-test-slave-hc1k krb5kdc[19034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430320, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:20.279287 19543 init.cc:377] Logged in from keytab as oryx/127.12.170.125@KRBTEST.COM (short username oryx)
I20260926 13:45:20.279767 19543 webserver.cc:460] Webserver started at http://127.12.170.125:46771/ using document root <none> and password file <none>
I20260926 13:45:20.279951 19543 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:20.280004 19543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:20.281280 19543 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:45:20.282362 19669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:20.282642 19543 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:45:20.282714 19543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-1/wal
uuid: "8189bfdb296f41d69b7e564da513093f"
format_stamp: "Formatted at 2026-09-26 13:45:20 on dist-test-slave-hc1k"
server_key: "e71556e69353e1eeff69a2e76236d141"
server_key_iv: "2b3585a4c32e6aa25045d0d265380eb2"
server_key_version: "encryptionkey@0"
I20260926 13:45:20.282786 19543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:20.312436 19543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:20.314375 19543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:20.319168 19543 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.125:40913
I20260926 13:45:20.319455 19721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.125:40913 every 8 connection(s)
I20260926 13:45:20.319886 19543 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-1/data/info.pb
I20260926 13:45:20.321228 19722 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:45:20.322026 19722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f: Bootstrap starting.
I20260926 13:45:20.327524 19722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:45:20.327776 19722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f: Bootstrap complete.
I20260926 13:45:20.328198 19722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: NON_VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: true } }
I20260926 13:45:20.328296 19722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8189bfdb296f41d69b7e564da513093f, State: Initialized, Role: LEARNER
I20260926 13:45:20.328390 19722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: NON_VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: true } }
I20260926 13:45:20.328648 19725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: NON_VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: true } } }
I20260926 13:45:20.328752 19725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [sys.catalog]: This master's current role is: LEARNER
I20260926 13:45:20.328783 19722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:45:20.331090 19736 catalog_manager.cc:1259] Loaded cluster ID: d34e7363d79146f9bc88ca2ccf28bda6
I20260926 13:45:20.331167 19736 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f: loading cluster ID for follower catalog manager: success
I20260926 13:45:20.332721 19736 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f: acquiring CA information for follower catalog manager: success
I20260926 13:45:20.333704 19736 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 13:45:20.351338 19651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:20.351558 19651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:20.351598 19651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:20.355484 19651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:45:20.355532 19651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:20.355549 19651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:20.355564 19651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:45:20.355580 19651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:45:20.360556 19651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:42035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:33065,127.12.170.125:40913,127.12.170.124:39363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.124.keytab
--principal=oryx/127.12.170.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.124:39363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.170.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:20.361481 19651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:20.362432 19651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:20.370036 19742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:20.370050 19744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:20.370090 19741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:20.370116 19651 server_base.cc:1034] running on GCE node
I20260926 13:45:20.370806 19651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:20.371831 19651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:20.372991 19651 hybrid_clock.cc:648] HybridClock initialized: now 1790430320372972 us; error 33 us; skew 500 ppm
Sep 26 13:45:20 dist-test-slave-hc1k krb5kdc[19034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430320, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:20.375432 19651 init.cc:377] Logged in from keytab as oryx/127.12.170.124@KRBTEST.COM (short username oryx)
I20260926 13:45:20.376288 19651 webserver.cc:460] Webserver started at http://127.12.170.124:40989/ using document root <none> and password file <none>
I20260926 13:45:20.376809 19651 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:20.376873 19651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:20.377051 19651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:20.378944 19651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-2/data/instance:
uuid: "4651c1b3f0a1489b8b9d564073bfe663"
format_stamp: "Formatted at 2026-09-26 13:45:20 on dist-test-slave-hc1k"
server_key: "1ea3879e3a32af0f8ec668c6014cd9e3"
server_key_iv: "7f5a9857436ef0c164bc94d091c03af7"
server_key_version: "encryptionkey@0"
I20260926 13:45:20.379520 19651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-2/wal/instance:
uuid: "4651c1b3f0a1489b8b9d564073bfe663"
format_stamp: "Formatted at 2026-09-26 13:45:20 on dist-test-slave-hc1k"
server_key: "1ea3879e3a32af0f8ec668c6014cd9e3"
server_key_iv: "7f5a9857436ef0c164bc94d091c03af7"
server_key_version: "encryptionkey@0"
I20260926 13:45:20.383173 19651 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:45:20.385608 19751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:20.386761 19651 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 13:45:20.386874 19651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-2/wal
uuid: "4651c1b3f0a1489b8b9d564073bfe663"
format_stamp: "Formatted at 2026-09-26 13:45:20 on dist-test-slave-hc1k"
server_key: "1ea3879e3a32af0f8ec668c6014cd9e3"
server_key_iv: "7f5a9857436ef0c164bc94d091c03af7"
server_key_version: "encryptionkey@0"
I20260926 13:45:20.386962 19651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:20.412708 19651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:20.415372 19651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:20.415535 19651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:20.422184 19651 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.124:39363
I20260926 13:45:20.422405 19803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.124:39363 every 8 connection(s)
I20260926 13:45:20.423089 19651 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-2/data/info.pb
I20260926 13:45:20.424045 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 19651
I20260926 13:45:20.426046 19804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:20.430361 19804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:45:20 dist-test-slave-hc1k krb5kdc[19034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.170.124@KRBTEST.COM
Sep 26 13:45:20 dist-test-slave-hc1k krb5kdc[19034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430320, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.124@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM
I20260926 13:45:20.438757 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.425471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49818 (local address 127.12.170.124:39363)
0926 13:45:20.425850 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:20.425867 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:20.425907 (+ 40us) server_negotiation.cc:407] Connection header received
0926 13:45:20.426717 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:20.426738 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:20.427132 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:20.427476 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:20.428135 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.429601 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.430519 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.431165 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.431311 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.433886 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:20.433893 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:20.433904 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:20.433933 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:20.435702 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.436248 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.436254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.436259 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.436428 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.436727 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.436731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.436732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.437142 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:20.437315 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:20.437571 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 13:45:20.437865 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"spinlock_wait_cycles":1280,"thread_start_us":105,"threads_started":1}
I20260926 13:45:20.440755 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.432212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48598 (local address 127.12.170.126:33065)
0926 13:45:20.432344 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:20.432349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:20.432522 (+ 173us) server_negotiation.cc:407] Connection header received
0926 13:45:20.432644 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:20.432647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:20.432698 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:20.432775 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:20.433519 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.434178 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.435165 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.435485 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.435605 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.437879 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:20.437883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:20.437885 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:20.437910 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:20.439228 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.439617 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.439620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.439621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.439725 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.440000 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.440002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.440003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.440150 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:20.440214 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:20.440513 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 13:45:20.440614 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 13:45:20.440788 19807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.432060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.0.0.1:48598)
0926 13:45:20.432407 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:20.432452 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:20.432558 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:20.432831 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:20.432837 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:20.433248 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:20.433413 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.433421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.434295 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.434298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.435057 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.435066 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.435622 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.435626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.435777 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.436262 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:20.436301 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:20.437748 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:20.439328 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.439335 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.439339 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.439530 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.439825 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.439828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.439830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.439918 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.440231 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:20.440236 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:20.440342 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 13:45:20.440468 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 13:45:20.440619 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"spinlock_wait_cycles":768,"thread_start_us":110,"threads_started":1}
I20260926 13:45:20.441696 19804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:45:20 dist-test-slave-hc1k krb5kdc[19034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430320, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.124@KRBTEST.COM for oryx/127.12.170.125@KRBTEST.COM
I20260926 13:45:20.448724 19807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.442058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:40913 (local address 127.0.0.1:60616)
0926 13:45:20.442336 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:20.442347 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:20.442400 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:20.442589 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:20.442591 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:20.442697 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:20.442775 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.442779 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.443644 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.443672 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.444245 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.444254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.444684 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.444686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.444753 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.445121 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:20.445132 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:20.446170 (+ 1038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:20.447630 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.447632 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.447634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.447863 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.448077 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.448079 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.448080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.448140 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.448407 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:20.448409 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:20.448472 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 13:45:20.448539 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 13:45:20.448620 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223}
I20260926 13:45:20.448788 19808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.442167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60616 (local address 127.12.170.125:40913)
0926 13:45:20.442389 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:20.442392 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:20.442404 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:45:20.442468 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:20.442470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:20.442507 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:20.442575 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:20.442899 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.443530 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.444378 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.444602 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.444674 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.446290 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:20.446293 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:20.446295 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:20.446317 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:20.447557 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.447938 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.447940 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.447941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.448013 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.448196 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.448198 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.448199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.448345 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:20.448416 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:20.448580 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 13:45:20.448667 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":66,"threads_started":1}
I20260926 13:45:20.449397 19804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.124" port: 39363 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:45:20 dist-test-slave-hc1k krb5kdc[19034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430320, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.124@KRBTEST.COM for oryx/127.12.170.124@KRBTEST.COM
I20260926 13:45:20.455679 19807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.449777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:39363 (local address 127.0.0.1:49826)
0926 13:45:20.449863 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:20.449875 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:20.449939 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:20.450087 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:20.450088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:20.450194 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:20.450253 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.450257 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.451198 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.451200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.451701 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.451709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.452101 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.452103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.452153 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.452500 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:20.452511 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:20.453473 (+ 962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:20.454651 (+ 1178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.454653 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.454655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.454819 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.455022 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.455024 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.455025 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.455083 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.455326 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:20.455328 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:20.455388 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 13:45:20.455455 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 13:45:20.455540 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260926 13:45:20.455758 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.449830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49826 (local address 127.12.170.124:39363)
0926 13:45:20.449942 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:20.449945 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:20.449956 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:45:20.449985 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:20.449987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:20.450016 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:20.450071 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:20.450323 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.451113 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.451797 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.452029 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.452099 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.453571 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:20.453573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:20.453575 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:20.453594 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:20.454572 (+ 978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.454904 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.454906 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.454907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.454965 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.455137 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.455139 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.455140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.455268 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:20.455330 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:20.455494 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 13:45:20.455585 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":59}
I20260926 13:45:20.458176 19804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663: Bootstrap starting.
I20260926 13:45:20.460357 19804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:20.461012 19804 log.cc:824] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:20.462716 19804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663: No bootstrap required, opened a new log
I20260926 13:45:20.465575 19804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 39363 } }
I20260926 13:45:20.465766 19804 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:20.465822 19804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4651c1b3f0a1489b8b9d564073bfe663, State: Initialized, Role: FOLLOWER
I20260926 13:45:20.466207 19804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 39363 } }
I20260926 13:45:20.466585 19810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 39363 } } }
I20260926 13:45:20.466689 19810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:45:20.467252 19804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 13:45:20.471386 19821 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:45:20.471462 19821 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:45:20.474246 19651 master_runner.cc:428] Detected that this master 4651c1b3f0a1489b8b9d564073bfe663 is joining an existing cluster
I20260926 13:45:20.474293 19651 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.170.124:39363
I20260926 13:45:20.484643 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.476504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49830 (local address 127.12.170.124:39363)
0926 13:45:20.476593 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:20.476597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:20.476804 (+ 207us) server_negotiation.cc:407] Connection header received
0926 13:45:20.476929 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:20.476932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:20.476965 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:20.477007 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:20.477675 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.478424 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.479809 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.480130 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.480209 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.481294 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:20.481297 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:20.481298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:20.481351 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:20.482877 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.483328 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.483330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.483332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.483412 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.483678 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.483681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.483682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.483883 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:20.483987 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:20.484358 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 13:45:20.484504 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":27}
I20260926 13:45:20.484658 19829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.476466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:39363 (local address 127.0.0.1:49830)
0926 13:45:20.476736 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:20.476748 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:20.476849 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:20.477116 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:20.477119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:20.477398 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:20.477552 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.477560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.478519 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.478523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.479694 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.479704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.480228 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.480232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.480331 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.480732 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:20.480746 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:20.481143 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:20.482975 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.482979 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.482981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.483247 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.483489 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.483492 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.483494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.483580 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.484007 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:20.484010 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:20.484171 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 13:45:20.484276 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 13:45:20.484449 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":86,"thread_start_us":73,"threads_started":1}
I20260926 13:45:20.491891 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 19141
I20260926 13:45:20.497980 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.476757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48602 (local address 127.12.170.126:33065)
0926 13:45:20.476864 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:20.476868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:20.476878 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:45:20.476911 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:20.476914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:20.476951 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:20.477011 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:20.477652 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.478447 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.479769 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.480109 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.480205 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.493533 (+ 13328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:20.493537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:20.493540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:20.493567 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:20.496241 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.496779 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.496783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.496784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.496864 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.497125 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.497128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.497130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.497334 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:20.497398 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:20.497717 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 13:45:20.497827 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:45:20.498445 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:33411
--local_ip_for_outbound_sockets=127.12.170.65
--tserver_master_addrs=127.12.170.126:33065
--webserver_port=40739
--webserver_interface=127.12.170.65
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.65.keytab
--principal=oryx/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:42035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.170.126:33065,127.12.170.125:40913
--tserver_master_addrs=127.12.170.126:33065,127.12.170.125:40913,127.12.170.124:39363 with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:45:20.498764 19827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.476086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.0.0.1:48602)
0926 13:45:20.476344 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:20.476357 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:20.476435 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:20.477113 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:20.477116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:20.477425 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:20.477543 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.477549 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.478540 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.478542 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.479645 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.479681 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.480311 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.480314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.480396 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.492942 (+ 12546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:20.492956 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:20.493380 (+ 424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:20.496366 (+ 2986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.496369 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.496371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.496675 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.496964 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.496967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.496969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.497041 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.497464 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:20.497467 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:20.497548 (+ 81us) client_negotiation.cc:769] Sending connection context
0926 13:45:20.498435 (+ 887us) client_negotiation.cc:240] Negotiation successful
0926 13:45:20.498566 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":71,"thread_start_us":93,"threads_started":1}
I20260926 13:45:20.500180 19808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.476732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60630 (local address 127.12.170.125:40913)
0926 13:45:20.476832 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:20.476836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:20.476857 (+ 21us) server_negotiation.cc:407] Connection header received
0926 13:45:20.476894 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:20.476897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:20.476941 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:20.477011 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:20.493105 (+ 16094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.493914 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.495074 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.495706 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.495797 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.497012 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:20.497015 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:20.497017 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:20.497042 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:20.498512 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.498990 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.498993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.498995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.499078 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.499329 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.499332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.499334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.499520 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:20.499602 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:20.499919 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 13:45:20.500033 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 13:45:20.500630 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.476123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:40913 (local address 127.0.0.1:60630)
0926 13:45:20.476440 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:20.476453 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:20.476534 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:20.477113 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:20.477116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:20.492857 (+ 15741us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:20.492967 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.492974 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.494049 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.494053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.494948 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.494957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.495920 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.495924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.496014 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.496419 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:20.496433 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:20.496870 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:20.498636 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.498639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.498640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.498875 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.499161 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.499164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.499165 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.499245 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.499644 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:20.499668 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:20.499749 (+ 81us) client_negotiation.cc:769] Sending connection context
0926 13:45:20.500346 (+ 597us) client_negotiation.cc:240] Negotiation successful
0926 13:45:20.500468 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":91,"threads_started":1}
Sep 26 13:45:20 dist-test-slave-hc1k krb5kdc[19034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430319, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.126@KRBTEST.COM for oryx/127.12.170.124@KRBTEST.COM
I20260926 13:45:20.511914 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.503767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49840 (local address 127.12.170.124:39363)
0926 13:45:20.504008 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:20.504011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:20.504019 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:45:20.504066 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:20.504068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:20.504103 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:20.504156 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:20.504586 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.505295 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.506352 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.506662 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.506758 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.508921 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:20.508925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:20.508928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:20.508951 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:20.510415 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.510844 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.510847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.510849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.510920 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.511154 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.511157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.511159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.511346 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:20.511432 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:20.511684 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 13:45:20.511782 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260926 13:45:20.511925 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.503711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:39363 (local address 127.0.0.1:49840)
0926 13:45:20.503821 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:20.503835 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:20.503903 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:20.504194 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:20.504197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:20.504378 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:20.504471 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.504477 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.505392 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.505396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.506239 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.506248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.506773 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.506777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.506870 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.507290 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:20.507303 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:20.508787 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:20.510518 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.510521 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.510524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.510752 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.510995 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.510998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.511000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.511081 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.511429 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:20.511431 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:20.511519 (+ 88us) client_negotiation.cc:769] Sending connection context
0926 13:45:20.511611 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 13:45:20.511758 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 13:45:20.513008 19079 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.170.124:39363: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663"
member_type: NON_VOTER
last_known_addr {
host: "127.12.170.124"
port: 39363
}
attrs {
promote: true
}
}
dest_uuid: "0f4afcedc19e4050b7b7320081502c13"
cas_config_opid_index: 5
I20260926 13:45:20.513597 19079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: NON_VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: true } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: NON_VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: true } }
I20260926 13:45:20.514763 19640 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4651c1b3f0a1489b8b9d564073bfe663 (127.12.170.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: NON_VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: true } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: NON_VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: true } } }
I20260926 13:45:20.516032 19778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:20.517390 19778 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [term 1 FOLLOWER]: Refusing update from remote peer 0f4afcedc19e4050b7b7320081502c13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 13:45:20.518635 19640 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.170.124:39363
I20260926 13:45:20.518797 19640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0f4afcedc19e4050b7b7320081502c13" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: NON_VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: true } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: NON_VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: true } } }
I20260926 13:45:20.518896 19640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:20.519302 19639 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: NON_VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:20.520049 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.514889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:40913 (local address 127.0.0.1:60632)
0926 13:45:20.515892 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:20.515905 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:20.515976 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:20.516336 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:20.516339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:20.516353 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:20.516422 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.516427 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.517388 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.517392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.518604 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.518614 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.519672 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.519676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.519764 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.519805 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 13:45:20.519872 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 13:45:20.519932 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":933}
I20260926 13:45:20.520354 19808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.515085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60632 (local address 127.12.170.125:40913)
0926 13:45:20.515299 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:20.515303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:20.516048 (+ 745us) server_negotiation.cc:407] Connection header received
0926 13:45:20.516095 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:20.516097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:20.516150 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:20.516224 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:20.516528 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.517255 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.518735 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.519352 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.519455 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.519594 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:20.520171 (+ 577us) server_negotiation.cc:299] Negotiation successful
0926 13:45:20.520236 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260926 13:45:20.520622 19651 master.cc:537] Master@127.12.170.124:39363 shutting down...
I20260926 13:45:20.520787 19696 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [term 1 LEARNER]: Refusing update from remote peer 0f4afcedc19e4050b7b7320081502c13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 13:45:20.521430 19639 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: NON_VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 13:45:20.522437 19696 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4651c1b3f0a1489b8b9d564073bfe663 (127.12.170.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: NON_VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: true } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: NON_VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: true } } }
I20260926 13:45:20.523999 19832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f4afcedc19e4050b7b7320081502c13" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: NON_VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: true } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: NON_VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: true } } }
I20260926 13:45:20.524151 19832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [sys.catalog]: This master's current role is: LEARNER
I20260926 13:45:20.524245 19725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f4afcedc19e4050b7b7320081502c13. Latest consensus state: current_term: 1 leader_uuid: "0f4afcedc19e4050b7b7320081502c13" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: NON_VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: true } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: NON_VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: true } } }
I20260926 13:45:20.524355 19725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [sys.catalog]: This master's current role is: LEARNER
I20260926 13:45:20.524978 19639 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13: attempting to promote NON_VOTER 8189bfdb296f41d69b7e564da513093f to VOTER
I20260926 13:45:20.525331 19639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: NON_VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: true } }
I20260926 13:45:20.526435 19696 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [term 1 LEARNER]: Refusing update from remote peer 0f4afcedc19e4050b7b7320081502c13: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 13:45:20.526834 19640 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260926 13:45:20.526912 19065 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 13:45:20.527727 19065 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 -> Peer 4651c1b3f0a1489b8b9d564073bfe663 (127.12.170.124:39363): Couldn't send request to peer 4651c1b3f0a1489b8b9d564073bfe663. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 13:45:20.528558 19640 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8189bfdb296f41d69b7e564da513093f (127.12.170.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: NON_VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: true } } }
I20260926 13:45:20.529450 19778 raft_consensus.cc:858] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 39363 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: NON_VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: NON_VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: true } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: NON_VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: true } }
W20260926 13:45:20.529541 19778 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 39363 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: NON_VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: NON_VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: true } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: NON_VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: true } }
I20260926 13:45:20.529772 19640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0f4afcedc19e4050b7b7320081502c13" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: NON_VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: true } } }
I20260926 13:45:20.529865 19640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:20.529978 19640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f4afcedc19e4050b7b7320081502c13" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: NON_VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: true } } }
I20260926 13:45:20.530043 19640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:20.530725 19696 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8189bfdb296f41d69b7e564da513093f (127.12.170.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: NON_VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: true } } }
I20260926 13:45:20.531113 19778 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8189bfdb296f41d69b7e564da513093f (127.12.170.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: NON_VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: true } } }
I20260926 13:45:20.532177 19725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f4afcedc19e4050b7b7320081502c13" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: NON_VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: true } } }
I20260926 13:45:20.532274 19725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:45:20.533025 19810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f4afcedc19e4050b7b7320081502c13. Latest consensus state: current_term: 1 leader_uuid: "0f4afcedc19e4050b7b7320081502c13" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: NON_VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: true } } }
I20260926 13:45:20.533128 19810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 13:45:20.533205 19810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f4afcedc19e4050b7b7320081502c13" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: NON_VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: true } } }
I20260926 13:45:20.533267 19810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 13:45:20.533320 19810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f4afcedc19e4050b7b7320081502c13" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: NON_VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: true } } }
I20260926 13:45:20.533375 19810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260926 13:45:20.535894 19834 mvcc.cc:205] Tried to move back new op lower bound from 7333602587920011264 to 7333602587771006976. Current Snapshot: MvccSnapshot[applied={T|T < 7333602587803504640}]
I20260926 13:45:20.541941 19651 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260926 13:45:20.542196 19651 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260926 13:45:20.542286 19651 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663: stopping tablet replica
I20260926 13:45:20.557102 19651 master.cc:552] Master@127.12.170.124:39363 shutdown complete.
I20260926 13:45:20.557157 19651 master_runner.cc:305] Clearing existing system tablet
I20260926 13:45:20.557379 19651 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:20.557430 19651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:20.558681 19651 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:45:20.559502 19842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:20.559713 19651 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:20.559842 19651 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:45:20.559876 19651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-2/wal
uuid: "4651c1b3f0a1489b8b9d564073bfe663"
format_stamp: "Formatted at 2026-09-26 13:45:20 on dist-test-slave-hc1k"
server_key: "1ea3879e3a32af0f8ec668c6014cd9e3"
server_key_iv: "7f5a9857436ef0c164bc94d091c03af7"
server_key_version: "encryptionkey@0"
I20260926 13:45:20.561517 19651 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:20.563731 19651 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 13:45:20.563812 19651 log.cc:1196] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 13:45:20.564088 19651 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663: Deleting consensus metadata
I20260926 13:45:20.564244 19651 master_runner.cc:315] Copying system tablet from 127.12.170.126:33065
I20260926 13:45:20.565301 19651 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663: tablet copy: Beginning tablet copy session from remote peer at address 127.12.170.126:33065
I20260926 13:45:20.574355 19848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.565925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.0.0.1:48614)
0926 13:45:20.566207 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:20.566220 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:20.566328 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:20.566753 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:20.566757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:20.566956 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:20.567104 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.567112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.568217 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.568221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.569249 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.569259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.569828 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.569832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.569920 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.570456 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:20.570471 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:20.570870 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:20.572719 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.572723 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.572725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.572960 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.573240 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.573243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.573245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.573347 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.573777 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:20.573780 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:20.573877 (+ 97us) client_negotiation.cc:769] Sending connection context
0926 13:45:20.574041 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 13:45:20.574175 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":75,"threads_started":1}
I20260926 13:45:20.574397 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.566019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.12.170.126:33065)
0926 13:45:20.566136 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:20.566140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:20.566296 (+ 156us) server_negotiation.cc:407] Connection header received
0926 13:45:20.566439 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:20.566442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:20.566509 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:20.566584 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:20.567251 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.568108 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.569374 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.569703 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.569826 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.571010 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:20.571014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:20.571017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:20.571042 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:20.572608 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.573054 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.573058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.573060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.573148 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.573433 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.573436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.573438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.573680 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:20.573777 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:20.574103 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 13:45:20.574214 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 13:45:20.575121 19099 tablet_copy_service.cc:140] P 0f4afcedc19e4050b7b7320081502c13: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4651c1b3f0a1489b8b9d564073bfe663 ({username='oryx', principal='oryx/127.12.170.124@KRBTEST.COM'} at 127.0.0.1:48614)
I20260926 13:45:20.575217 19099 tablet_copy_service.cc:161] P 0f4afcedc19e4050b7b7320081502c13: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4651c1b3f0a1489b8b9d564073bfe663 at {username='oryx', principal='oryx/127.12.170.124@KRBTEST.COM'} at 127.0.0.1:48614: session id = 4651c1b3f0a1489b8b9d564073bfe663-00000000000000000000000000000000
I20260926 13:45:20.576320 19099 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 13:45:20.577543 19651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 13:45:20.579772 19651 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663: tablet copy: Starting download of 0 data blocks...
I20260926 13:45:20.579941 19651 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663: tablet copy: Starting download of 1 WAL segments...
I20260926 13:45:20.581322 19651 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 13:45:20.583132 19099 tablet_copy_service.cc:342] P 0f4afcedc19e4050b7b7320081502c13: Request end of tablet copy session 4651c1b3f0a1489b8b9d564073bfe663-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.170.124@KRBTEST.COM'} at 127.0.0.1:48614
I20260926 13:45:20.583209 19099 tablet_copy_service.cc:433] P 0f4afcedc19e4050b7b7320081502c13: ending tablet copy session 4651c1b3f0a1489b8b9d564073bfe663-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4651c1b3f0a1489b8b9d564073bfe663
I20260926 13:45:20.590521 19651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:20.592299 19852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:20.592294 19849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:20.592546 19850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:20.592746 19651 server_base.cc:1034] running on GCE node
I20260926 13:45:20.592888 19651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:20.593158 19651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:20.594290 19651 hybrid_clock.cc:648] HybridClock initialized: now 1790430320594272 us; error 36 us; skew 500 ppm
Sep 26 13:45:20 dist-test-slave-hc1k krb5kdc[19034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430320, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:20.595913 19651 init.cc:377] Logged in from keytab as oryx/127.12.170.124@KRBTEST.COM (short username oryx)
I20260926 13:45:20.596376 19651 webserver.cc:460] Webserver started at http://127.12.170.124:43775/ using document root <none> and password file <none>
I20260926 13:45:20.596558 19651 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:20.596607 19651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:20.598047 19651 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:45:20.599023 19859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:20.599294 19651 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:45:20.599380 19651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-2/wal
uuid: "4651c1b3f0a1489b8b9d564073bfe663"
format_stamp: "Formatted at 2026-09-26 13:45:20 on dist-test-slave-hc1k"
server_key: "1ea3879e3a32af0f8ec668c6014cd9e3"
server_key_iv: "7f5a9857436ef0c164bc94d091c03af7"
server_key_version: "encryptionkey@0"
I20260926 13:45:20.599462 19651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:20.617951 19651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:20.620008 19651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:20.624575 19651 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.124:39363
I20260926 13:45:20.624599 19914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.124:39363 every 8 connection(s)
I20260926 13:45:20.625285 19651 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/master-2/data/info.pb
I20260926 13:45:20.626926 19915 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:45:20.627970 19915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663: Bootstrap starting.
W20260926 13:45:20.628736 19830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:20.629097 19830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:20.629168 19830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:20.634431 19830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:20.634495 19830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:20.634579 19830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:45:20.635511 19915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:45:20.635769 19915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663: Bootstrap complete.
I20260926 13:45:20.636274 19915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: NON_VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: true } }
I20260926 13:45:20.636394 19915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4651c1b3f0a1489b8b9d564073bfe663, State: Initialized, Role: LEARNER
I20260926 13:45:20.636490 19915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: NON_VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: true } }
I20260926 13:45:20.636757 19918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: NON_VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: true } } }
I20260926 13:45:20.636865 19918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [sys.catalog]: This master's current role is: LEARNER
I20260926 13:45:20.636889 19915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:45:20.639320 19929 catalog_manager.cc:1259] Loaded cluster ID: d34e7363d79146f9bc88ca2ccf28bda6
I20260926 13:45:20.639369 19929 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663: loading cluster ID for follower catalog manager: success
I20260926 13:45:20.640971 19929 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663: acquiring CA information for follower catalog manager: success
I20260926 13:45:20.640977 19830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:42035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.65.keytab
--principal=oryx/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:33411
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=40739
--tserver_master_addrs=127.12.170.126:33065,127.12.170.125:40913,127.12.170.124:39363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:20.641913 19830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:20.642009 19929 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:45:20.642853 19830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:45:20.651670 19830 server_base.cc:1034] running on GCE node
W20260926 13:45:20.651685 19931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:20.651682 19932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:20.651748 19934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:20.652396 19830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:20.652941 19830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:20.654129 19830 hybrid_clock.cc:648] HybridClock initialized: now 1790430320654112 us; error 24 us; skew 500 ppm
Sep 26 13:45:20 dist-test-slave-hc1k krb5kdc[19034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430320, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:20.656649 19830 init.cc:377] Logged in from keytab as oryx/127.12.170.65@KRBTEST.COM (short username oryx)
I20260926 13:45:20.657444 19830 webserver.cc:460] Webserver started at http://127.12.170.65:40739/ using document root <none> and password file <none>
I20260926 13:45:20.657994 19830 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:20.658079 19830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:20.661761 19830 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 13:45:20.663592 19941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:20.664767 19830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:20.664862 19830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "144cccc67d584072b304beeb25acad33"
format_stamp: "Formatted at 2026-09-26 13:45:19 on dist-test-slave-hc1k"
server_key: "7ebf4d9612cd05608fafb5dda261e91f"
server_key_iv: "6729515311ab76e4243329716e1a4d78"
server_key_version: "encryptionkey@0"
I20260926 13:45:20.665378 19830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:20.679543 19830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:20.682701 19830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:20.682865 19830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:20.683425 19830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:20.684263 19830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:20.684320 19830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:20.684389 19830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:20.684417 19830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:20.692348 19830 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:33411
I20260926 13:45:20.692395 20054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:33411 every 8 connection(s)
I20260926 13:45:20.693264 19830 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:45:20.698658 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 19830
I20260926 13:45:20.698829 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 19275
Sep 26 13:45:20 dist-test-slave-hc1k krb5kdc[19034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430320, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM
Sep 26 13:45:20 dist-test-slave-hc1k krb5kdc[19034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430320, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.124@KRBTEST.COM
Sep 26 13:45:20 dist-test-slave-hc1k krb5kdc[19034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430320, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.125@KRBTEST.COM
I20260926 13:45:20.705988 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:33105
--local_ip_for_outbound_sockets=127.12.170.66
--tserver_master_addrs=127.12.170.126:33065
--webserver_port=40589
--webserver_interface=127.12.170.66
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.66.keytab
--principal=oryx/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:42035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.170.126:33065,127.12.170.125:40913
--tserver_master_addrs=127.12.170.126:33065,127.12.170.125:40913,127.12.170.124:39363 with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:45:20.709141 19808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.694846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:34825 (local address 127.12.170.125:40913)
0926 13:45:20.694970 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:20.694975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:20.695871 (+ 896us) server_negotiation.cc:407] Connection header received
0926 13:45:20.696934 (+ 1063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:20.696937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:20.696995 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:20.697065 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:20.698409 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.699172 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.701395 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.701739 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.701808 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.705841 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:20.705845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:20.705848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:20.705875 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:20.707403 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.707885 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.707889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.707891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.707976 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.708260 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.708264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.708266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.708482 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:20.708553 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:20.708872 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 13:45:20.708989 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 13:45:20.709177 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.696367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:42895 (local address 127.12.170.124:39363)
0926 13:45:20.696576 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:20.696579 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:20.696592 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:45:20.697187 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:20.697191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:20.697239 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:20.697324 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:20.698452 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.699504 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.701020 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.701366 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.701453 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.705208 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:20.705212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:20.705214 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:20.705242 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:20.706874 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.707449 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.707453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.707454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.707540 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.707929 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.707932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.707934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.708140 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:20.708220 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:20.708916 (+ 696us) server_negotiation.cc:299] Negotiation successful
0926 13:45:20.709026 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":70,"threads_started":1}
I20260926 13:45:20.709966 20059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.695266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:40913 (local address 127.12.170.65:34825)
0926 13:45:20.695724 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:20.695775 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:20.696748 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:20.697234 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:20.697242 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:20.697737 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:20.698281 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.698295 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.699367 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.699370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.701268 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.701279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.701966 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.701969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.702066 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.702837 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:20.702862 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:20.705720 (+ 2858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:20.707508 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.707511 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.707513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.707775 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.708080 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.708082 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.708084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.708162 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.708603 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:20.708606 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:20.708688 (+ 82us) client_negotiation.cc:769] Sending connection context
0926 13:45:20.709325 (+ 637us) client_negotiation.cc:240] Negotiation successful
0926 13:45:20.709455 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":929,"spinlock_wait_cycles":9600,"thread_start_us":94,"threads_started":1}
I20260926 13:45:20.710213 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.696209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:39363 (local address 127.12.170.65:42895)
0926 13:45:20.696449 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:20.696462 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:20.696754 (+ 292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:20.697349 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:20.697352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:20.697742 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:20.698279 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.698296 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.699624 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.699627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.700891 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.700902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.701605 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.701609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.701939 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.702720 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:20.702732 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:20.705054 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:20.706993 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.706998 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.707011 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.707325 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.707636 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.707639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.707641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.707827 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.708268 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:20.708274 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:20.708564 (+ 290us) client_negotiation.cc:769] Sending connection context
0926 13:45:20.708742 (+ 178us) client_negotiation.cc:240] Negotiation successful
0926 13:45:20.709030 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":204,"spinlock_wait_cycles":13952,"thread_start_us":76,"threads_started":1}
I20260926 13:45:20.710579 20060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.695266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.12.170.65:46173)
0926 13:45:20.695724 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:20.695774 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:20.696747 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:20.697234 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:20.697243 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:20.697737 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:20.698279 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.698296 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.699365 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.699368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.700891 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.700902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.701605 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.701609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.701940 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.702676 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:20.702697 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:20.706421 (+ 3724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:20.708514 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.708518 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.708520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.708743 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.709039 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.709042 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.709044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.709118 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.709889 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:20.709892 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:20.709973 (+ 81us) client_negotiation.cc:769] Sending connection context
0926 13:45:20.710057 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 13:45:20.710165 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":197,"spinlock_wait_cycles":70016,"thread_start_us":104,"threads_started":1}
I20260926 13:45:20.710743 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.694865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:46173 (local address 127.12.170.126:33065)
0926 13:45:20.695000 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:20.695005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:20.695881 (+ 876us) server_negotiation.cc:407] Connection header received
0926 13:45:20.697045 (+ 1164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:20.697048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:20.697097 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:20.697162 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:20.698437 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.699225 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.701020 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.701366 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.701439 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.706573 (+ 5134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:20.706577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:20.706580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:20.706607 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:20.708406 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.708852 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.708856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.708857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.708939 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.709533 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.709537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.709539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.709749 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:20.709846 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:20.710493 (+ 647us) server_negotiation.cc:299] Negotiation successful
0926 13:45:20.710608 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 13:45:20.711174 20055 heartbeater.cc:347] Connected to a master server at 127.12.170.124:39363
I20260926 13:45:20.711242 20057 heartbeater.cc:347] Connected to a master server at 127.12.170.125:40913
I20260926 13:45:20.711393 20055 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:20.711392 20057 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:20.711932 20057 heartbeater.cc:511] Master 127.12.170.125:40913 requested a full tablet report, sending...
I20260926 13:45:20.711932 20055 heartbeater.cc:511] Master 127.12.170.124:39363 requested a full tablet report, sending...
I20260926 13:45:20.712414 20056 heartbeater.cc:347] Connected to a master server at 127.12.170.126:33065
I20260926 13:45:20.712482 20056 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:20.712612 20056 heartbeater.cc:511] Master 127.12.170.126:33065 requested a full tablet report, sending...
I20260926 13:45:20.713299 19079 ts_manager.cc:194] Re-registered known tserver with Master: 144cccc67d584072b304beeb25acad33 (127.12.170.65:33411)
I20260926 13:45:20.713416 19686 ts_manager.cc:194] Registered new tserver with Master: 144cccc67d584072b304beeb25acad33 (127.12.170.65:33411)
I20260926 13:45:20.713510 19876 ts_manager.cc:194] Registered new tserver with Master: 144cccc67d584072b304beeb25acad33 (127.12.170.65:33411)
I20260926 13:45:20.714052 19079 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.170.65@KRBTEST.COM'} at 127.12.170.65:46173
W20260926 13:45:20.812208 20063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:20.812423 20063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:20.812460 20063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:20.816354 20063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:20.816394 20063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:20.816447 20063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:45:20.821303 20063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:42035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.66.keytab
--principal=oryx/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:33105
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=40589
--tserver_master_addrs=127.12.170.126:33065,127.12.170.125:40913,127.12.170.124:39363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:20.822134 20063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:20.822966 20063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:20.831573 20068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:20.831610 20069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:20.831626 20071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:20.831861 20063 server_base.cc:1034] running on GCE node
I20260926 13:45:20.832191 20063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:20.832733 20063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:20.833887 20063 hybrid_clock.cc:648] HybridClock initialized: now 1790430320833870 us; error 27 us; skew 500 ppm
Sep 26 13:45:20 dist-test-slave-hc1k krb5kdc[19034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430320, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:20.836382 20063 init.cc:377] Logged in from keytab as oryx/127.12.170.66@KRBTEST.COM (short username oryx)
I20260926 13:45:20.837210 20063 webserver.cc:460] Webserver started at http://127.12.170.66:40589/ using document root <none> and password file <none>
I20260926 13:45:20.837771 20063 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:20.837841 20063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:20.841406 20063 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 13:45:20.843300 20078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:20.844441 20063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:20.844549 20063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "5378a400ff1e490fb8c805c8465a4f37"
format_stamp: "Formatted at 2026-09-26 13:45:19 on dist-test-slave-hc1k"
server_key: "01c531701f41a8036003ace7ebbc93a6"
server_key_iv: "7c8ac42d47ca893b5a659dad63ea4fe0"
server_key_version: "encryptionkey@0"
I20260926 13:45:20.845016 20063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:20.873379 20063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:20.875962 20063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:20.876116 20063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:20.876657 20063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:20.877437 20063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:20.877483 20063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:20.877532 20063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:20.877558 20063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:20.885818 20063 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:33105
I20260926 13:45:20.885864 20191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:33105 every 8 connection(s)
I20260926 13:45:20.886719 20063 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:45:20.894349 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 20063
I20260926 13:45:20.894526 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 19409
Sep 26 13:45:20 dist-test-slave-hc1k krb5kdc[19034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430320, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM
Sep 26 13:45:20 dist-test-slave-hc1k krb5kdc[19034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430320, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.124@KRBTEST.COM
Sep 26 13:45:20 dist-test-slave-hc1k krb5kdc[19034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430320, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.125@KRBTEST.COM
I20260926 13:45:20.902206 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:34977
--local_ip_for_outbound_sockets=127.12.170.67
--tserver_master_addrs=127.12.170.126:33065
--webserver_port=43189
--webserver_interface=127.12.170.67
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.67.keytab
--principal=oryx/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:42035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.170.126:33065,127.12.170.125:40913
--tserver_master_addrs=127.12.170.126:33065,127.12.170.125:40913,127.12.170.124:39363 with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:45:20.902731 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.890354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:36479 (local address 127.12.170.126:33065)
0926 13:45:20.890482 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:20.890486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:20.890625 (+ 139us) server_negotiation.cc:407] Connection header received
0926 13:45:20.890748 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:20.890751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:20.890793 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:20.890851 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:20.892254 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.893063 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.894394 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.894699 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.894758 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.898565 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:20.898569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:20.898572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:20.898598 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:20.900459 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.901071 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.901074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.901076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.901163 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.901516 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.901537 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.901538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.901733 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:20.901801 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:20.902473 (+ 672us) server_negotiation.cc:299] Negotiation successful
0926 13:45:20.902583 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:45:20.902732 19808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.889996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:34163 (local address 127.12.170.125:40913)
0926 13:45:20.890114 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:20.890118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:20.890290 (+ 172us) server_negotiation.cc:407] Connection header received
0926 13:45:20.890970 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:20.890973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:20.891022 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:20.891094 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:20.892280 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.892788 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.894192 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.894513 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.894578 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.898851 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:20.898853 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:20.898855 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:20.898877 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:20.900418 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.901098 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.901101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.901103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.901176 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.901539 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.901543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.901545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.901751 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:20.901818 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:20.902461 (+ 643us) server_negotiation.cc:299] Negotiation successful
0926 13:45:20.902580 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 13:45:20.902913 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.888329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:35977 (local address 127.12.170.124:39363)
0926 13:45:20.888485 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:20.888488 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:20.889662 (+ 1174us) server_negotiation.cc:407] Connection header received
0926 13:45:20.890728 (+ 1066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:20.890731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:20.890783 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:20.890854 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:20.892284 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.893067 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.894044 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.894345 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.894444 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.898385 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:20.898389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:20.898392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:20.898423 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:20.900291 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.901041 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.901045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.901047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.901130 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:20.901554 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:20.901557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:20.901559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:20.901751 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:20.901818 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:20.902661 (+ 843us) server_negotiation.cc:299] Negotiation successful
0926 13:45:20.902778 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 13:45:20.904348 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.889891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:40913 (local address 127.12.170.66:34163)
0926 13:45:20.890126 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:20.890140 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:20.890521 (+ 381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:20.891120 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:20.891123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:20.891462 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:20.892129 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.892141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.892925 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.892927 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.894065 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.894075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.895313 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.895316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.895385 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.896500 (+ 1115us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:20.896513 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:20.898733 (+ 2220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:20.900574 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.900578 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.900580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.901005 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.901268 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.901271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.901273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.901414 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.901902 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:20.901905 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:20.902187 (+ 282us) client_negotiation.cc:769] Sending connection context
0926 13:45:20.902924 (+ 737us) client_negotiation.cc:240] Negotiation successful
0926 13:45:20.903286 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":64,"spinlock_wait_cycles":30336,"thread_start_us":72,"threads_started":1}
I20260926 13:45:20.904366 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.890249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.12.170.66:36479)
0926 13:45:20.890553 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:20.890564 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:20.890646 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:20.890997 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:20.891005 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:20.891454 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:20.892124 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.892140 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.893463 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.893467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.894284 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.894291 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.894968 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.894972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.895065 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.896236 (+ 1171us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:20.896249 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:20.898421 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:20.900575 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.900578 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.900580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.900962 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.901264 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.901267 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.901269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.901417 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.901881 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:20.901887 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:20.902189 (+ 302us) client_negotiation.cc:769] Sending connection context
0926 13:45:20.902918 (+ 729us) client_negotiation.cc:240] Negotiation successful
0926 13:45:20.903377 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":283,"spinlock_wait_cycles":27776,"thread_start_us":120,"threads_started":1}
I20260926 13:45:20.904956 20196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:20.888698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:39363 (local address 127.12.170.66:35977)
0926 13:45:20.889524 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:20.889564 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:20.890518 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:20.890997 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:20.891003 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:20.891454 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:20.892132 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.892141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.893170 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.893173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.893937 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:20.893946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:20.894596 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:20.894600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:20.894777 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:20.896097 (+ 1320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:20.896115 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:20.898210 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:20.900430 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.900436 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.900450 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.900933 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.901235 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:20.901238 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:20.901240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:20.901414 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:20.901933 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:20.901935 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:20.902187 (+ 252us) client_negotiation.cc:769] Sending connection context
0926 13:45:20.903505 (+ 1318us) client_negotiation.cc:240] Negotiation successful
0926 13:45:20.903627 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":182,"spinlock_wait_cycles":18048,"thread_start_us":480,"threads_started":1}
I20260926 13:45:20.905496 20194 heartbeater.cc:347] Connected to a master server at 127.12.170.125:40913
I20260926 13:45:20.905522 20193 heartbeater.cc:347] Connected to a master server at 127.12.170.126:33065
I20260926 13:45:20.905715 20194 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:20.905714 20193 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:20.906279 20193 heartbeater.cc:511] Master 127.12.170.126:33065 requested a full tablet report, sending...
I20260926 13:45:20.906277 20194 heartbeater.cc:511] Master 127.12.170.125:40913 requested a full tablet report, sending...
I20260926 13:45:20.907279 19079 ts_manager.cc:194] Re-registered known tserver with Master: 5378a400ff1e490fb8c805c8465a4f37 (127.12.170.66:33105)
I20260926 13:45:20.907310 19686 ts_manager.cc:194] Registered new tserver with Master: 5378a400ff1e490fb8c805c8465a4f37 (127.12.170.66:33105)
I20260926 13:45:20.907706 20192 heartbeater.cc:347] Connected to a master server at 127.12.170.124:39363
I20260926 13:45:20.907778 20192 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:20.907918 20192 heartbeater.cc:511] Master 127.12.170.124:39363 requested a full tablet report, sending...
I20260926 13:45:20.908043 19079 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.170.66@KRBTEST.COM'} at 127.12.170.66:36479
I20260926 13:45:20.908524 19876 ts_manager.cc:194] Registered new tserver with Master: 5378a400ff1e490fb8c805c8465a4f37 (127.12.170.66:33105)
W20260926 13:45:21.007292 20199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:21.007551 20199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:21.007592 20199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:21.011441 20199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:21.011489 20199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:21.011544 20199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:45:21.016505 20199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:42035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.67.keytab
--principal=oryx/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:34977
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=43189
--tserver_master_addrs=127.12.170.126:33065,127.12.170.125:40913,127.12.170.124:39363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:21.017376 20199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:21.018270 20199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:21.027043 20207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:21.027034 20204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:21.027016 20205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:21.027094 20199 server_base.cc:1034] running on GCE node
I20260926 13:45:21.027541 20199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:21.028234 20199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:21.029389 20199 hybrid_clock.cc:648] HybridClock initialized: now 1790430321029365 us; error 37 us; skew 500 ppm
Sep 26 13:45:21 dist-test-slave-hc1k krb5kdc[19034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430321, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:21.031984 20199 init.cc:377] Logged in from keytab as oryx/127.12.170.67@KRBTEST.COM (short username oryx)
I20260926 13:45:21.032756 20199 webserver.cc:460] Webserver started at http://127.12.170.67:43189/ using document root <none> and password file <none>
I20260926 13:45:21.033221 20199 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:21.033284 20199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:21.036752 20199 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 13:45:21.038697 20214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:21.039801 20199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 13:45:21.039896 20199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "875eb6152ee6435683b49d64a8986403"
format_stamp: "Formatted at 2026-09-26 13:45:19 on dist-test-slave-hc1k"
server_key: "8f80528cb9c6e86d44674edb91c6fca1"
server_key_iv: "24cece39732052991783e12b77f0e68e"
server_key_version: "encryptionkey@0"
I20260926 13:45:21.040356 20199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:21.059486 20199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:21.061933 20199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:21.062083 20199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:21.062570 20199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:21.063304 20199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:21.063347 20199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:21.063395 20199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:21.063423 20199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:21.071520 20199 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:34977
I20260926 13:45:21.071545 20327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:34977 every 8 connection(s)
I20260926 13:45:21.072384 20199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:45:21.078428 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 20199
I20260926 13:45:21.082069 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:21.075937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:39363 (local address 127.0.0.1:49856)
0926 13:45:21.076133 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:21.076146 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:21.076234 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:21.076592 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:21.076595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:21.076610 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:21.076700 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:21.076708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.077723 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.077726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:21.080604 (+ 2878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:21.080616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.081637 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.081641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:21.081728 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:21.081750 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 13:45:21.081831 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 13:45:21.081919 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":55,"threads_started":1}
I20260926 13:45:21.082109 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:21.076009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49856 (local address 127.12.170.124:39363)
0926 13:45:21.076216 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:21.076220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:21.076233 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:45:21.076337 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:21.076340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:21.076404 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:21.076491 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:21.076826 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.077586 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.080725 (+ 3139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.081323 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.081422 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:21.081573 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:21.081848 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 13:45:21.081935 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":64,"threads_started":1}
I20260926 13:45:21.082903 19886 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
Sep 26 13:45:21 dist-test-slave-hc1k krb5kdc[19034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430321, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.124@KRBTEST.COM
I20260926 13:45:21.083323 19918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f4afcedc19e4050b7b7320081502c13. Latest consensus state: current_term: 1 leader_uuid: "0f4afcedc19e4050b7b7320081502c13" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: NON_VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: true } } }
I20260926 13:45:21.083451 19918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [sys.catalog]: This master's current role is: LEARNER
Sep 26 13:45:21 dist-test-slave-hc1k krb5kdc[19034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430321, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM
Sep 26 13:45:21 dist-test-slave-hc1k krb5kdc[19034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430321, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.125@KRBTEST.COM
I20260926 13:45:21.090026 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:21.074014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:51027 (local address 127.12.170.126:33065)
0926 13:45:21.074128 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:21.074132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:21.075145 (+ 1013us) server_negotiation.cc:407] Connection header received
0926 13:45:21.076377 (+ 1232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:21.076380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:21.076421 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:21.076479 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:21.077938 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.078730 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.080472 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.080806 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.080871 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:21.086772 (+ 5901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:21.086776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:21.086778 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:21.086806 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:21.088390 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:21.088839 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:21.088842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:21.088844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:21.088924 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:21.089199 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:21.089202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:21.089204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:21.089395 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:21.089468 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:21.089777 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 13:45:21.089880 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:45:21.090157 20333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:21.074505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:39363 (local address 127.12.170.67:47739)
0926 13:45:21.074992 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:21.075046 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:21.076196 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:21.076783 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:21.076787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:21.077127 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:21.077792 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:21.077801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.078766 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.078769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:21.080242 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:21.080253 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.081032 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.081036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:21.081301 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:21.082033 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:21.082072 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:21.083772 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:21.085682 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:21.085688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:21.085703 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:21.086023 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:21.087722 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:21.087726 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:21.087727 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:21.087900 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:21.088353 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:21.088359 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:21.088648 (+ 289us) client_negotiation.cc:769] Sending connection context
0926 13:45:21.088834 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 13:45:21.089159 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":269,"spinlock_wait_cycles":72704,"thread_start_us":126,"threads_started":1}
I20260926 13:45:21.090472 19808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:21.075816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:44851 (local address 127.12.170.125:40913)
0926 13:45:21.075931 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:21.075936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:21.075950 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:45:21.076562 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:21.076565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:21.076616 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:21.076664 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:21.077938 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.078726 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.080371 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.080705 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.080775 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:21.087143 (+ 6368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:21.087147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:21.087149 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:21.087174 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:21.088853 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:21.089303 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:21.089307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:21.089309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:21.089388 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:21.089653 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:21.089656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:21.089658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:21.089854 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:21.089928 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:21.090235 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 13:45:21.090338 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 13:45:21.090842 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:21.074506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.12.170.67:51027)
0926 13:45:21.074992 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:21.075046 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:21.076196 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:21.076623 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:21.076631 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:21.077129 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:21.077782 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:21.077800 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.078906 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.078909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:21.080376 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:21.080381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.081076 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.081079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:21.081663 (+ 584us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:21.083951 (+ 2288us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:21.083961 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:21.086259 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:21.088486 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:21.088489 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:21.088492 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:21.088740 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:21.089024 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:21.089027 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:21.089029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:21.089111 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:21.089522 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:21.089525 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:21.089604 (+ 79us) client_negotiation.cc:769] Sending connection context
0926 13:45:21.090532 (+ 928us) client_negotiation.cc:240] Negotiation successful
0926 13:45:21.090659 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":319,"spinlock_wait_cycles":8064,"thread_start_us":126,"threads_started":1}
I20260926 13:45:21.090883 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:21.074014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:47739 (local address 127.12.170.124:39363)
0926 13:45:21.074128 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:21.074132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:21.075142 (+ 1010us) server_negotiation.cc:407] Connection header received
0926 13:45:21.076585 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:21.076588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:21.076621 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:21.076683 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:21.077920 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.078596 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.080509 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.080802 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.080884 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:21.083902 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:21.083906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:21.083909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:21.083929 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:21.085560 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:21.087517 (+ 1957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:21.087521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:21.087523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:21.087605 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:21.088011 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:21.088015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:21.088017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:21.088202 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:21.088296 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:21.090636 (+ 2340us) server_negotiation.cc:299] Negotiation successful
0926 13:45:21.090750 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:45:21.091754 20328 heartbeater.cc:347] Connected to a master server at 127.12.170.124:39363
I20260926 13:45:21.091986 20328 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:21.092013 20334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:21.075493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:40913 (local address 127.12.170.67:44851)
0926 13:45:21.075753 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:21.075765 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:21.076460 (+ 695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:21.076823 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:21.076825 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:21.077125 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:21.077801 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:21.077808 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.078867 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.078871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:21.080242 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:21.080253 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.081028 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.081032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:21.081294 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:21.084917 (+ 3623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:21.084944 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:21.086617 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:21.088950 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:21.088953 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:21.088955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:21.089198 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:21.089483 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:21.089486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:21.089488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:21.089563 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:21.089982 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:21.089984 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:21.090062 (+ 78us) client_negotiation.cc:769] Sending connection context
0926 13:45:21.090830 (+ 768us) client_negotiation.cc:240] Negotiation successful
0926 13:45:21.090953 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":276,"spinlock_wait_cycles":13056,"thread_start_us":74,"threads_started":1}
I20260926 13:45:21.092494 20328 heartbeater.cc:511] Master 127.12.170.124:39363 requested a full tablet report, sending...
I20260926 13:45:21.092988 20329 heartbeater.cc:347] Connected to a master server at 127.12.170.126:33065
I20260926 13:45:21.093057 20329 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:21.093202 20329 heartbeater.cc:511] Master 127.12.170.126:33065 requested a full tablet report, sending...
I20260926 13:45:21.093379 20330 heartbeater.cc:347] Connected to a master server at 127.12.170.125:40913
I20260926 13:45:21.093441 20330 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:21.093565 20330 heartbeater.cc:511] Master 127.12.170.125:40913 requested a full tablet report, sending...
I20260926 13:45:21.094431 19079 ts_manager.cc:194] Re-registered known tserver with Master: 875eb6152ee6435683b49d64a8986403 (127.12.170.67:34977)
I20260926 13:45:21.095121 19079 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.170.67@KRBTEST.COM'} at 127.12.170.67:51027
I20260926 13:45:21.095638 19686 ts_manager.cc:194] Registered new tserver with Master: 875eb6152ee6435683b49d64a8986403 (127.12.170.67:34977)
I20260926 13:45:21.095953 19876 ts_manager.cc:194] Registered new tserver with Master: 875eb6152ee6435683b49d64a8986403 (127.12.170.67:34977)
I20260926 13:45:21.097890 20342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:21.081389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48624 (local address 127.12.170.126:33065)
0926 13:45:21.083684 (+ 2295us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:21.083688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:21.083699 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:45:21.083729 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:21.083732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:21.083769 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:21.083837 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:21.088724 (+ 4887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.089429 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.092648 (+ 3219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.092969 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.093061 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:21.094510 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:21.094514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:21.094517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:21.094543 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:21.096116 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:21.096557 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:21.096560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:21.096562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:21.096643 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:21.096928 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:21.096931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:21.096933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:21.097104 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:21.097216 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:21.097687 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 13:45:21.097774 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2215,"thread_start_us":72,"threads_started":1}
I20260926 13:45:21.098634 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:21.082988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49860 (local address 127.12.170.124:39363)
0926 13:45:21.083084 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:21.083088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:21.084879 (+ 1791us) server_negotiation.cc:407] Connection header received
0926 13:45:21.084914 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:21.084918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:21.084955 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:21.085017 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:21.086250 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.086935 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.089695 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.090004 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.090091 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:21.093486 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:21.093489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:21.093492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:21.093516 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:21.095811 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:21.097219 (+ 1408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:21.097222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:21.097224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:21.097311 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:21.097579 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:21.097582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:21.097583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:21.097803 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:21.097927 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:21.098380 (+ 453us) server_negotiation.cc:299] Negotiation successful
0926 13:45:21.098500 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 13:45:21.099310 20341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:21.080363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60638 (local address 127.12.170.125:40913)
0926 13:45:21.083232 (+ 2869us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:21.083236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:21.084443 (+ 1207us) server_negotiation.cc:407] Connection header received
0926 13:45:21.084551 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:21.084554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:21.084597 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:21.084672 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:21.085862 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.086666 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.090930 (+ 4264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.091281 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.091366 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:21.095937 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:21.095940 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:21.095942 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:21.095964 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:21.097514 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:21.097924 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:21.097927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:21.097929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:21.098005 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:21.098251 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:21.098254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:21.098256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:21.098463 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:21.098579 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:21.099026 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 13:45:21.099140 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2805,"thread_start_us":73,"threads_started":1}
I20260926 13:45:21.100769 19079 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:45:21.103451 19079 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:45:21.105741 19639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f4afcedc19e4050b7b7320081502c13" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: NON_VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: true } } }
I20260926 13:45:21.105857 19639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:21.107595 20347 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13: attempting to promote NON_VOTER 4651c1b3f0a1489b8b9d564073bfe663 to VOTER
I20260926 13:45:21.108603 20347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: false } }
I20260926 13:45:21.109372 19639 raft_consensus.cc:1013] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [term 1 LEADER]: attempt to promote peer 4651c1b3f0a1489b8b9d564073bfe663: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260926 13:45:21.109660 19696 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [term 1 FOLLOWER]: Refusing update from remote peer 0f4afcedc19e4050b7b7320081502c13: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 13:45:21.109678 19886 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [term 1 LEARNER]: Refusing update from remote peer 0f4afcedc19e4050b7b7320081502c13: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 13:45:21.110059 19639 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 13:45:21.111379 20349 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 4651c1b3f0a1489b8b9d564073bfe663 (127.12.170.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: false } } }
I20260926 13:45:21.111917 19639 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 13:45:21.111887 19886 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 4651c1b3f0a1489b8b9d564073bfe663 (127.12.170.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: false } } }
I20260926 13:45:21.112663 19696 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 4651c1b3f0a1489b8b9d564073bfe663 (127.12.170.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: false } } }
I20260926 13:45:21.112771 20349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0f4afcedc19e4050b7b7320081502c13" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: false } } }
I20260926 13:45:21.112857 20349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:21.112965 20349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f4afcedc19e4050b7b7320081502c13" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: false } } }
I20260926 13:45:21.113034 20349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:21.114037 20357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f4afcedc19e4050b7b7320081502c13" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: false } } }
I20260926 13:45:21.114149 20357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:45:21.114111 19918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f4afcedc19e4050b7b7320081502c13" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: false } } }
I20260926 13:45:21.114213 19918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:45:21.114854 20348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f4afcedc19e4050b7b7320081502c13" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: false } } }
I20260926 13:45:21.114946 20348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13 [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:21.122956 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:21.117727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:34977 (local address 127.0.0.1:48766)
0926 13:45:21.118172 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:21.118188 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:21.118257 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:21.119530 (+ 1273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:21.119533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:21.119548 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:21.119629 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:21.119635 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.120808 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.120812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:21.121578 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:21.121587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.122593 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.122596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:21.122676 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:21.122696 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 13:45:21.122767 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 13:45:21.122846 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382}
I20260926 13:45:21.123001 20358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:21.117843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48766 (local address 127.12.170.67:34977)
0926 13:45:21.119166 (+ 1323us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:21.119178 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:21.119199 (+ 21us) server_negotiation.cc:407] Connection header received
0926 13:45:21.119265 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:21.119271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:21.119419 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:21.119544 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:21.119840 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.120699 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.121727 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.122488 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.122582 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:21.122719 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:21.122797 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 13:45:21.122888 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1224,"thread_start_us":104,"threads_started":1}
I20260926 13:45:21.123327 20360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:21.118208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38968 (local address 127.12.170.66:33105)
0926 13:45:21.118532 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:21.118544 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:21.118566 (+ 22us) server_negotiation.cc:407] Connection header received
0926 13:45:21.118640 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:21.118645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:21.118879 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:21.119039 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:21.119216 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.120357 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.121767 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.122568 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.122644 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:21.122761 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:21.123103 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 13:45:21.123198 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":129,"threads_started":1}
I20260926 13:45:21.123585 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:21.118465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:33411 (local address 127.0.0.1:53670)
0926 13:45:21.118853 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:21.118866 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:21.118938 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:21.119753 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:21.119756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:21.119773 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:21.119855 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:21.119861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.120938 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.120941 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:21.122017 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:21.122026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.123216 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.123219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:21.123313 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:21.123335 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 13:45:21.123405 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 13:45:21.123466 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":74,"threads_started":1}
I20260926 13:45:21.123677 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:21.118609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53670 (local address 127.12.170.65:33411)
0926 13:45:21.119323 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:21.119334 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:21.119354 (+ 20us) server_negotiation.cc:407] Connection header received
0926 13:45:21.119421 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:21.119426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:21.119608 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:21.119783 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:21.119942 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.120836 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.122146 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.123087 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.123209 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:21.123332 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:21.123430 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 13:45:21.123526 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":82,"threads_started":1}
I20260926 13:45:21.123937 20359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:21.118025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:33105 (local address 127.0.0.1:38968)
0926 13:45:21.118223 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:21.118235 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:21.118313 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:21.119009 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:21.119012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:21.119024 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:21.119119 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:21.119124 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.120465 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.120469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:21.121631 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:21.121640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.122864 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.122867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:21.122956 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:21.122977 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 13:45:21.123605 (+ 628us) client_negotiation.cc:240] Negotiation successful
0926 13:45:21.123808 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
I20260926 13:45:21.125913 20262 tablet_service.cc:1467] Processing CreateTablet for tablet 86901e00029143cb805e2f70be1dc963 (DEFAULT_TABLE table=test-table [id=1a65d36de39647849e3535a0f794b3dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:21.126246 19989 tablet_service.cc:1467] Processing CreateTablet for tablet 86901e00029143cb805e2f70be1dc963 (DEFAULT_TABLE table=test-table [id=1a65d36de39647849e3535a0f794b3dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:21.126246 20126 tablet_service.cc:1467] Processing CreateTablet for tablet 86901e00029143cb805e2f70be1dc963 (DEFAULT_TABLE table=test-table [id=1a65d36de39647849e3535a0f794b3dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:21.126783 20262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86901e00029143cb805e2f70be1dc963. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:21.127163 20126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86901e00029143cb805e2f70be1dc963. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:21.127163 19989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86901e00029143cb805e2f70be1dc963. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:21.132992 20363 tablet_bootstrap.cc:492] T 86901e00029143cb805e2f70be1dc963 P 875eb6152ee6435683b49d64a8986403: Bootstrap starting.
I20260926 13:45:21.134402 20365 tablet_bootstrap.cc:492] T 86901e00029143cb805e2f70be1dc963 P 144cccc67d584072b304beeb25acad33: Bootstrap starting.
I20260926 13:45:21.135313 20363 tablet_bootstrap.cc:654] T 86901e00029143cb805e2f70be1dc963 P 875eb6152ee6435683b49d64a8986403: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:21.135986 20364 tablet_bootstrap.cc:492] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37: Bootstrap starting.
I20260926 13:45:21.136168 20363 log.cc:824] T 86901e00029143cb805e2f70be1dc963 P 875eb6152ee6435683b49d64a8986403: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:21.136404 20365 tablet_bootstrap.cc:654] T 86901e00029143cb805e2f70be1dc963 P 144cccc67d584072b304beeb25acad33: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:21.137190 20365 log.cc:824] T 86901e00029143cb805e2f70be1dc963 P 144cccc67d584072b304beeb25acad33: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:21.138011 20364 tablet_bootstrap.cc:654] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:21.138266 20363 tablet_bootstrap.cc:492] T 86901e00029143cb805e2f70be1dc963 P 875eb6152ee6435683b49d64a8986403: No bootstrap required, opened a new log
I20260926 13:45:21.138447 20363 ts_tablet_manager.cc:1397] T 86901e00029143cb805e2f70be1dc963 P 875eb6152ee6435683b49d64a8986403: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 13:45:21.138792 20364 log.cc:824] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:21.139072 20365 tablet_bootstrap.cc:492] T 86901e00029143cb805e2f70be1dc963 P 144cccc67d584072b304beeb25acad33: No bootstrap required, opened a new log
I20260926 13:45:21.139242 20365 ts_tablet_manager.cc:1397] T 86901e00029143cb805e2f70be1dc963 P 144cccc67d584072b304beeb25acad33: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 13:45:21.140234 20364 tablet_bootstrap.cc:492] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37: No bootstrap required, opened a new log
I20260926 13:45:21.140399 20364 ts_tablet_manager.cc:1397] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 13:45:21.142771 20363 raft_consensus.cc:348] T 86901e00029143cb805e2f70be1dc963 P 875eb6152ee6435683b49d64a8986403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875eb6152ee6435683b49d64a8986403" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34977 } } peers { permanent_uuid: "5378a400ff1e490fb8c805c8465a4f37" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 33105 } } peers { permanent_uuid: "144cccc67d584072b304beeb25acad33" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 33411 } }
I20260926 13:45:21.143029 20363 raft_consensus.cc:374] T 86901e00029143cb805e2f70be1dc963 P 875eb6152ee6435683b49d64a8986403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:21.143126 20363 raft_consensus.cc:729] T 86901e00029143cb805e2f70be1dc963 P 875eb6152ee6435683b49d64a8986403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 875eb6152ee6435683b49d64a8986403, State: Initialized, Role: FOLLOWER
I20260926 13:45:21.143437 20365 raft_consensus.cc:348] T 86901e00029143cb805e2f70be1dc963 P 144cccc67d584072b304beeb25acad33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875eb6152ee6435683b49d64a8986403" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34977 } } peers { permanent_uuid: "5378a400ff1e490fb8c805c8465a4f37" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 33105 } } peers { permanent_uuid: "144cccc67d584072b304beeb25acad33" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 33411 } }
I20260926 13:45:21.143550 20363 consensus_queue.cc:260] T 86901e00029143cb805e2f70be1dc963 P 875eb6152ee6435683b49d64a8986403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875eb6152ee6435683b49d64a8986403" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34977 } } peers { permanent_uuid: "5378a400ff1e490fb8c805c8465a4f37" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 33105 } } peers { permanent_uuid: "144cccc67d584072b304beeb25acad33" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 33411 } }
I20260926 13:45:21.143755 20365 raft_consensus.cc:374] T 86901e00029143cb805e2f70be1dc963 P 144cccc67d584072b304beeb25acad33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:21.143821 20365 raft_consensus.cc:729] T 86901e00029143cb805e2f70be1dc963 P 144cccc67d584072b304beeb25acad33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 144cccc67d584072b304beeb25acad33, State: Initialized, Role: FOLLOWER
I20260926 13:45:21.143754 20364 raft_consensus.cc:348] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875eb6152ee6435683b49d64a8986403" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34977 } } peers { permanent_uuid: "5378a400ff1e490fb8c805c8465a4f37" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 33105 } } peers { permanent_uuid: "144cccc67d584072b304beeb25acad33" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 33411 } }
I20260926 13:45:21.143968 20364 raft_consensus.cc:374] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:21.144018 20364 raft_consensus.cc:729] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5378a400ff1e490fb8c805c8465a4f37, State: Initialized, Role: FOLLOWER
I20260926 13:45:21.144263 20365 consensus_queue.cc:260] T 86901e00029143cb805e2f70be1dc963 P 144cccc67d584072b304beeb25acad33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875eb6152ee6435683b49d64a8986403" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34977 } } peers { permanent_uuid: "5378a400ff1e490fb8c805c8465a4f37" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 33105 } } peers { permanent_uuid: "144cccc67d584072b304beeb25acad33" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 33411 } }
I20260926 13:45:21.144394 20364 consensus_queue.cc:260] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875eb6152ee6435683b49d64a8986403" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34977 } } peers { permanent_uuid: "5378a400ff1e490fb8c805c8465a4f37" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 33105 } } peers { permanent_uuid: "144cccc67d584072b304beeb25acad33" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 33411 } }
I20260926 13:45:21.144606 20363 ts_tablet_manager.cc:1428] T 86901e00029143cb805e2f70be1dc963 P 875eb6152ee6435683b49d64a8986403: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 13:45:21.145133 20365 ts_tablet_manager.cc:1428] T 86901e00029143cb805e2f70be1dc963 P 144cccc67d584072b304beeb25acad33: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 13:45:21.145293 20364 ts_tablet_manager.cc:1428] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 13:45:21.146029 20329 heartbeater.cc:503] Master 127.12.170.126:33065 was elected leader, sending a full tablet report...
I20260926 13:45:21.146747 20056 heartbeater.cc:503] Master 127.12.170.126:33065 was elected leader, sending a full tablet report...
I20260926 13:45:21.150111 20193 heartbeater.cc:503] Master 127.12.170.126:33065 was elected leader, sending a full tablet report...
W20260926 13:45:21.195077 20058 tablet.cc:2369] T 86901e00029143cb805e2f70be1dc963 P 144cccc67d584072b304beeb25acad33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:45:21.272639 20371 raft_consensus.cc:482] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:45:21.272819 20371 raft_consensus.cc:504] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875eb6152ee6435683b49d64a8986403" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34977 } } peers { permanent_uuid: "5378a400ff1e490fb8c805c8465a4f37" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 33105 } } peers { permanent_uuid: "144cccc67d584072b304beeb25acad33" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 33411 } }
I20260926 13:45:21.273902 20371 leader_election.cc:288] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 875eb6152ee6435683b49d64a8986403 (127.12.170.67:34977), 144cccc67d584072b304beeb25acad33 (127.12.170.65:33411)
I20260926 13:45:21.278445 20358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:21.274194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:51371 (local address 127.12.170.67:34977)
0926 13:45:21.274305 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:21.274309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:21.274397 (+ 88us) server_negotiation.cc:407] Connection header received
0926 13:45:21.274486 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:21.274489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:21.274533 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:21.274596 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:21.274815 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.275599 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.277237 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.277834 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.277956 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:21.278003 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:21.278257 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 13:45:21.278325 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:45:21.278445 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:21.274193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:37127 (local address 127.12.170.65:33411)
0926 13:45:21.274305 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:21.274309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:21.274326 (+ 17us) server_negotiation.cc:407] Connection header received
0926 13:45:21.274380 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:21.274384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:21.274433 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:21.274549 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:21.274821 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.275620 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.277239 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.277834 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.277951 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:21.278001 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:21.278257 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 13:45:21.278322 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:45:21.278507 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:21.274098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:34977 (local address 127.12.170.66:51371)
0926 13:45:21.274315 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:21.274326 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:21.274409 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:21.274606 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:21.274609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:21.274620 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:21.274704 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:21.274711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.275779 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.275782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:21.277118 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:21.277130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.277939 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.277943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:21.278113 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:21.278166 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 13:45:21.278238 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 13:45:21.278351 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":213}
I20260926 13:45:21.278507 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:21.274098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:33411 (local address 127.12.170.66:37127)
0926 13:45:21.274200 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:21.274213 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:21.274297 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:21.274560 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:21.274562 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:21.274577 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:21.274712 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:21.274717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.275729 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.275732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:21.277118 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:21.277130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.277939 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.277943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:21.278113 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:21.278167 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 13:45:21.278238 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 13:45:21.278351 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":175}
I20260926 13:45:21.279172 20282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86901e00029143cb805e2f70be1dc963" candidate_uuid: "5378a400ff1e490fb8c805c8465a4f37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875eb6152ee6435683b49d64a8986403" is_pre_election: true
I20260926 13:45:21.279175 20009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86901e00029143cb805e2f70be1dc963" candidate_uuid: "5378a400ff1e490fb8c805c8465a4f37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "144cccc67d584072b304beeb25acad33" is_pre_election: true
I20260926 13:45:21.279423 20282 raft_consensus.cc:2454] T 86901e00029143cb805e2f70be1dc963 P 875eb6152ee6435683b49d64a8986403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5378a400ff1e490fb8c805c8465a4f37 in term 0.
I20260926 13:45:21.279430 20009 raft_consensus.cc:2454] T 86901e00029143cb805e2f70be1dc963 P 144cccc67d584072b304beeb25acad33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5378a400ff1e490fb8c805c8465a4f37 in term 0.
I20260926 13:45:21.279829 20082 leader_election.cc:302] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5378a400ff1e490fb8c805c8465a4f37, 875eb6152ee6435683b49d64a8986403; no voters:
I20260926 13:45:21.280086 20371 raft_consensus.cc:2781] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:45:21.280158 20371 raft_consensus.cc:482] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:45:21.280189 20371 raft_consensus.cc:3037] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:21.281250 20371 raft_consensus.cc:504] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875eb6152ee6435683b49d64a8986403" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34977 } } peers { permanent_uuid: "5378a400ff1e490fb8c805c8465a4f37" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 33105 } } peers { permanent_uuid: "144cccc67d584072b304beeb25acad33" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 33411 } }
I20260926 13:45:21.281597 20371 leader_election.cc:288] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37 [CANDIDATE]: Term 1 election: Requested vote from peers 875eb6152ee6435683b49d64a8986403 (127.12.170.67:34977), 144cccc67d584072b304beeb25acad33 (127.12.170.65:33411)
I20260926 13:45:21.281898 20282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86901e00029143cb805e2f70be1dc963" candidate_uuid: "5378a400ff1e490fb8c805c8465a4f37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875eb6152ee6435683b49d64a8986403"
I20260926 13:45:21.281898 20009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86901e00029143cb805e2f70be1dc963" candidate_uuid: "5378a400ff1e490fb8c805c8465a4f37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "144cccc67d584072b304beeb25acad33"
I20260926 13:45:21.281999 20009 raft_consensus.cc:3037] T 86901e00029143cb805e2f70be1dc963 P 144cccc67d584072b304beeb25acad33 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:21.281999 20282 raft_consensus.cc:3037] T 86901e00029143cb805e2f70be1dc963 P 875eb6152ee6435683b49d64a8986403 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:21.282981 20009 raft_consensus.cc:2454] T 86901e00029143cb805e2f70be1dc963 P 144cccc67d584072b304beeb25acad33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5378a400ff1e490fb8c805c8465a4f37 in term 1.
I20260926 13:45:21.283012 20282 raft_consensus.cc:2454] T 86901e00029143cb805e2f70be1dc963 P 875eb6152ee6435683b49d64a8986403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5378a400ff1e490fb8c805c8465a4f37 in term 1.
I20260926 13:45:21.283245 20082 leader_election.cc:302] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5378a400ff1e490fb8c805c8465a4f37, 875eb6152ee6435683b49d64a8986403; no voters:
I20260926 13:45:21.283392 20371 raft_consensus.cc:2781] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:21.283565 20371 raft_consensus.cc:686] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37 [term 1 LEADER]: Becoming Leader. State: Replica: 5378a400ff1e490fb8c805c8465a4f37, State: Running, Role: LEADER
I20260926 13:45:21.283911 20371 consensus_queue.cc:237] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875eb6152ee6435683b49d64a8986403" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34977 } } peers { permanent_uuid: "5378a400ff1e490fb8c805c8465a4f37" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 33105 } } peers { permanent_uuid: "144cccc67d584072b304beeb25acad33" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 33411 } }
I20260926 13:45:21.287055 19078 catalog_manager.cc:5495] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5378a400ff1e490fb8c805c8465a4f37 (127.12.170.66). New cstate: current_term: 1 leader_uuid: "5378a400ff1e490fb8c805c8465a4f37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "875eb6152ee6435683b49d64a8986403" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5378a400ff1e490fb8c805c8465a4f37" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 33105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "144cccc67d584072b304beeb25acad33" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 33411 } health_report { overall_health: UNKNOWN } } }
I20260926 13:45:21.305493 20360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:21.302326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38978 (local address 127.12.170.66:33105)
0926 13:45:21.302422 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:21.302426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:21.302437 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:45:21.302548 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:21.302550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:21.302596 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:21.302677 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:45:21.302865 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.303404 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.304235 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.304443 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.304512 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:21.304760 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:45:21.305153 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:45:21.305252 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:21.305349 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 13:45:21.305395 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 13:45:21.310745 20009 raft_consensus.cc:1261] T 86901e00029143cb805e2f70be1dc963 P 144cccc67d584072b304beeb25acad33 [term 1 FOLLOWER]: Refusing update from remote peer 5378a400ff1e490fb8c805c8465a4f37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:21.310745 20282 raft_consensus.cc:1261] T 86901e00029143cb805e2f70be1dc963 P 875eb6152ee6435683b49d64a8986403 [term 1 FOLLOWER]: Refusing update from remote peer 5378a400ff1e490fb8c805c8465a4f37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:21.311403 20372 consensus_queue.cc:1035] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "144cccc67d584072b304beeb25acad33" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 33411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:21.311559 20371 consensus_queue.cc:1035] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875eb6152ee6435683b49d64a8986403" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:21.321818 20376 mvcc.cc:205] Tried to move back new op lower bound from 7333602596082544640 to 7333602595981103104. Current Snapshot: MvccSnapshot[applied={T|T < 7333602596082544640 or (T in {7333602596082544640})}]
I20260926 13:45:21.323055 20377 mvcc.cc:205] Tried to move back new op lower bound from 7333602596082544640 to 7333602595981103104. Current Snapshot: MvccSnapshot[applied={T|T < 7333602596082544640}]
I20260926 13:45:21.329975 19077 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:45:21.330127 19077 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:45:21.333725 19077 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f4afcedc19e4050b7b7320081502c13: Sending DeleteTablet for 3 replicas of tablet 86901e00029143cb805e2f70be1dc963
I20260926 13:45:21.334343 20262 tablet_service.cc:1514] Processing DeleteTablet for tablet 86901e00029143cb805e2f70be1dc963 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:21 UTC) from {username='oryx', principal='oryx/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:48766
I20260926 13:45:21.334717 19989 tablet_service.cc:1514] Processing DeleteTablet for tablet 86901e00029143cb805e2f70be1dc963 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:21 UTC) from {username='oryx', principal='oryx/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:53670
I20260926 13:45:21.335120 20386 tablet_replica.cc:330] T 86901e00029143cb805e2f70be1dc963 P 144cccc67d584072b304beeb25acad33: stopping tablet replica
I20260926 13:45:21.335549 20386 raft_consensus.cc:2229] T 86901e00029143cb805e2f70be1dc963 P 144cccc67d584072b304beeb25acad33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:45:21.336000 20386 raft_consensus.cc:2258] T 86901e00029143cb805e2f70be1dc963 P 144cccc67d584072b304beeb25acad33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:21.336211 20385 tablet_replica.cc:330] T 86901e00029143cb805e2f70be1dc963 P 875eb6152ee6435683b49d64a8986403: stopping tablet replica
I20260926 13:45:21.336546 20385 raft_consensus.cc:2229] T 86901e00029143cb805e2f70be1dc963 P 875eb6152ee6435683b49d64a8986403 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:45:21.336853 20385 raft_consensus.cc:2258] T 86901e00029143cb805e2f70be1dc963 P 875eb6152ee6435683b49d64a8986403 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:21.337755 20386 ts_tablet_manager.cc:1905] T 86901e00029143cb805e2f70be1dc963 P 144cccc67d584072b304beeb25acad33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:21.338376 20385 ts_tablet_manager.cc:1905] T 86901e00029143cb805e2f70be1dc963 P 875eb6152ee6435683b49d64a8986403: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:21.339848 20126 tablet_service.cc:1514] Processing DeleteTablet for tablet 86901e00029143cb805e2f70be1dc963 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:21 UTC) from {username='oryx', principal='oryx/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:38968
I20260926 13:45:21.340684 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 19047
I20260926 13:45:21.340700 20386 ts_tablet_manager.cc:1918] T 86901e00029143cb805e2f70be1dc963 P 144cccc67d584072b304beeb25acad33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:45:21.340793 20386 log.cc:1196] T 86901e00029143cb805e2f70be1dc963 P 144cccc67d584072b304beeb25acad33: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-0/wal/wals/86901e00029143cb805e2f70be1dc963
I20260926 13:45:21.341100 20386 ts_tablet_manager.cc:1939] T 86901e00029143cb805e2f70be1dc963 P 144cccc67d584072b304beeb25acad33: Deleting consensus metadata
I20260926 13:45:21.342223 20385 ts_tablet_manager.cc:1918] T 86901e00029143cb805e2f70be1dc963 P 875eb6152ee6435683b49d64a8986403: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:45:21.342312 20385 log.cc:1196] T 86901e00029143cb805e2f70be1dc963 P 875eb6152ee6435683b49d64a8986403: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/86901e00029143cb805e2f70be1dc963
I20260926 13:45:21.342633 20385 ts_tablet_manager.cc:1939] T 86901e00029143cb805e2f70be1dc963 P 875eb6152ee6435683b49d64a8986403: Deleting consensus metadata
I20260926 13:45:21.343704 20388 tablet_replica.cc:330] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37: stopping tablet replica
I20260926 13:45:21.343964 20388 raft_consensus.cc:2229] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:45:21.344254 20388 raft_consensus.cc:2258] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:21.345491 20388 ts_tablet_manager.cc:1905] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 13:45:21.346889 19944 connection.cc:535] server connection from 127.0.0.1:53670 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 13:45:21.347121 20215 connection.cc:535] server connection from 127.0.0.1:48766 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 13:45:21.347900 20388 ts_tablet_manager.cc:1918] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:45:21.347985 20388 log.cc:1196] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/wal/wals/86901e00029143cb805e2f70be1dc963
I20260926 13:45:21.348290 20388 ts_tablet_manager.cc:1939] T 86901e00029143cb805e2f70be1dc963 P 5378a400ff1e490fb8c805c8465a4f37: Deleting consensus metadata
W20260926 13:45:21.350458 20079 connection.cc:441] server connection from 127.0.0.1:38968 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38968 (request call id 1) could send its response
I20260926 13:45:21.357664 20341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:21.336887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60646 (local address 127.12.170.125:40913)
0926 13:45:21.337143 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:21.337148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:21.348752 (+ 11604us) server_negotiation.cc:407] Connection header received
0926 13:45:21.348955 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:21.348959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:21.349050 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:21.349129 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:21.350210 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.351048 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.352141 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.352516 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.352613 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:21.354353 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:21.354357 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:21.354360 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:21.354384 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:21.355973 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:21.356428 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:21.356431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:21.356433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:21.356512 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:21.356785 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:21.356788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:21.356790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:21.356976 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:21.357069 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:21.357401 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 13:45:21.357513 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260926 13:45:21.361332 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:21.352270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49884 (local address 127.12.170.124:39363)
0926 13:45:21.352393 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:21.352398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:21.352411 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:45:21.352451 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:21.352455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:21.352505 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:21.352574 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:21.353620 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.354340 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.355427 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.355823 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.355905 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:21.357630 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:21.357635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:21.357638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:21.357666 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:21.359313 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:21.359883 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:21.359887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:21.359889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:21.359980 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:21.360321 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:21.360325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:21.360327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:21.360550 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:21.360659 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:21.361050 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 13:45:21.361165 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 13:45:21.363058 19808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:21.350889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60656 (local address 127.12.170.125:40913)
0926 13:45:21.351003 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:21.351007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:21.352343 (+ 1336us) server_negotiation.cc:407] Connection header received
0926 13:45:21.352513 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:21.352516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:21.352559 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:21.352609 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:21.354250 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.354984 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.356100 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.356417 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.356500 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:21.359240 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:21.359244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:21.359247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:21.359287 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:21.361013 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:21.361727 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:21.361731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:21.361733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:21.361826 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:21.362146 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:21.362149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:21.362151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:21.362371 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:21.362469 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:21.362823 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 13:45:21.362922 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 13:45:21.366545 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:21.337016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49874 (local address 127.12.170.124:39363)
0926 13:45:21.337126 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:21.337130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:21.358157 (+ 21027us) server_negotiation.cc:407] Connection header received
0926 13:45:21.358301 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:21.358304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:21.358356 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:21.358402 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:21.359070 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.359862 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.360944 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:21.361273 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:21.361351 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:21.363238 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:21.363242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:21.363245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:21.363269 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:21.364870 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:21.365361 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:21.365364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:21.365366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:21.365446 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:21.365721 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:21.365724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:21.365726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:21.365899 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:21.365988 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:21.366301 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 13:45:21.366394 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 13:45:22.324211 20396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.323839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.12.170.66:50047)
0926 13:45:22.324056 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.324130 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":72,"threads_started":1}
W20260926 13:45:22.324496 20193 heartbeater.cc:644] Failed to heartbeat to 127.12.170.126:33065 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
I20260926 13:45:22.326002 20397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.325234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.12.170.65:43233)
0926 13:45:22.325828 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.325914 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":66,"threads_started":1}
W20260926 13:45:22.326284 20056 heartbeater.cc:644] Failed to heartbeat to 127.12.170.126:33065 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
I20260926 13:45:22.327873 20398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.327526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.12.170.67:57391)
0926 13:45:22.327740 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.327802 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":56,"threads_started":1}
W20260926 13:45:22.328109 20329 heartbeater.cc:644] Failed to heartbeat to 127.12.170.126:33065 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
I20260926 13:45:22.834556 20399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0f4afcedc19e4050b7b7320081502c13)
I20260926 13:45:22.834729 20399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: false } }
I20260926 13:45:22.835687 20399 leader_election.cc:288] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f4afcedc19e4050b7b7320081502c13 (127.12.170.126:33065), 8189bfdb296f41d69b7e564da513093f (127.12.170.125:40913)
I20260926 13:45:22.836278 20400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.835841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.0.0.1:49072)
0926 13:45:22.836112 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.836186 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":56,"threads_started":1}
W20260926 13:45:22.836463 19863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
I20260926 13:45:22.837235 20400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.837002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.0.0.1:49076)
0926 13:45:22.837125 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.837175 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 13:45:22.837338 19863 leader_election.cc:334] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0f4afcedc19e4050b7b7320081502c13 (127.12.170.126:33065): Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
I20260926 13:45:22.839414 20404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0f4afcedc19e4050b7b7320081502c13)
I20260926 13:45:22.839583 20404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: false } }
I20260926 13:45:22.839735 20401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.835886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:40913 (local address 127.0.0.1:60658)
0926 13:45:22.836203 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.836218 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:22.836330 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:22.836638 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:22.836641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:22.836662 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:22.836771 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:22.836778 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.837764 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.837766 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:22.838686 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:22.838696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.839325 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.839329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:22.839414 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:22.839442 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 13:45:22.839514 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 13:45:22.839586 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":49,"threads_started":1}
I20260926 13:45:22.839766 20402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.835964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60658 (local address 127.12.170.125:40913)
0926 13:45:22.836248 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:22.836252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:22.836281 (+ 29us) server_negotiation.cc:407] Connection header received
0926 13:45:22.836461 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:22.836465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:22.836515 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:22.836600 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:22.836883 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.837644 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.838784 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.839198 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.839321 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:22.839374 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:22.839549 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 13:45:22.839622 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":63,"threads_started":1}
I20260926 13:45:22.840260 19696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4651c1b3f0a1489b8b9d564073bfe663" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8189bfdb296f41d69b7e564da513093f" is_pre_election: true
I20260926 13:45:22.840441 19696 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4651c1b3f0a1489b8b9d564073bfe663 in term 1.
I20260926 13:45:22.840713 20404 leader_election.cc:288] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f4afcedc19e4050b7b7320081502c13 (127.12.170.126:33065), 4651c1b3f0a1489b8b9d564073bfe663 (127.12.170.124:39363)
I20260926 13:45:22.840762 19862 leader_election.cc:302] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4651c1b3f0a1489b8b9d564073bfe663, 8189bfdb296f41d69b7e564da513093f; no voters: 0f4afcedc19e4050b7b7320081502c13
I20260926 13:45:22.841013 20399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 13:45:22.841064 20399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0f4afcedc19e4050b7b7320081502c13)
I20260926 13:45:22.841100 20399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [term 1 FOLLOWER]: Advancing to term 2
I20260926 13:45:22.841109 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.840778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.0.0.1:49078)
0926 13:45:22.840967 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.841033 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":62,"threads_started":1}
W20260926 13:45:22.841243 19673 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
I20260926 13:45:22.842025 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.841785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.0.0.1:49088)
0926 13:45:22.841927 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.841971 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260926 13:45:22.842121 19673 leader_election.cc:334] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0f4afcedc19e4050b7b7320081502c13 (127.12.170.126:33065): Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
I20260926 13:45:22.842375 20399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: false } }
I20260926 13:45:22.842662 20399 leader_election.cc:288] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [CANDIDATE]: Term 2 election: Requested vote from peers 0f4afcedc19e4050b7b7320081502c13 (127.12.170.126:33065), 8189bfdb296f41d69b7e564da513093f (127.12.170.125:40913)
I20260926 13:45:22.842927 20401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.842766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.0.0.1:49094)
0926 13:45:22.842838 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.842881 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260926 13:45:22.842965 19696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4651c1b3f0a1489b8b9d564073bfe663" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8189bfdb296f41d69b7e564da513093f"
I20260926 13:45:22.843075 19696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [term 1 FOLLOWER]: Advancing to term 2
I20260926 13:45:22.843495 20401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.843335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.0.0.1:49110)
0926 13:45:22.843408 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.843450 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260926 13:45:22.843588 19863 leader_election.cc:334] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0f4afcedc19e4050b7b7320081502c13 (127.12.170.126:33065): Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
I20260926 13:45:22.844228 19696 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4651c1b3f0a1489b8b9d564073bfe663 in term 2.
I20260926 13:45:22.844538 19862 leader_election.cc:302] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4651c1b3f0a1489b8b9d564073bfe663, 8189bfdb296f41d69b7e564da513093f; no voters: 0f4afcedc19e4050b7b7320081502c13
I20260926 13:45:22.844754 20399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 13:45:22.844961 20399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [term 2 LEADER]: Becoming Leader. State: Replica: 4651c1b3f0a1489b8b9d564073bfe663, State: Running, Role: LEADER
I20260926 13:45:22.845082 20406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.840858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:39363 (local address 127.0.0.1:49896)
0926 13:45:22.841219 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.841232 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:22.841329 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:22.841771 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:22.841775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:22.841790 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:22.841881 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:22.841886 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.842739 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.842743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:22.843951 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:22.843961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.844734 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.844738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:22.844843 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:22.844867 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 13:45:22.844930 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 13:45:22.844982 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":58,"threads_started":1}
I20260926 13:45:22.845139 20407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.841161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49896 (local address 127.12.170.124:39363)
0926 13:45:22.841538 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:22.841546 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:22.841558 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:45:22.841595 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:22.841598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:22.841639 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:22.841761 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:22.841970 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.842623 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.844092 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.844635 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.844744 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:22.844802 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:22.844958 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 13:45:22.845024 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":109,"threads_started":1}
I20260926 13:45:22.845359 20399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: false } }
I20260926 13:45:22.845424 19886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8189bfdb296f41d69b7e564da513093f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4651c1b3f0a1489b8b9d564073bfe663" is_pre_election: true
I20260926 13:45:22.845876 19672 leader_election.cc:302] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8189bfdb296f41d69b7e564da513093f; no voters: 0f4afcedc19e4050b7b7320081502c13, 4651c1b3f0a1489b8b9d564073bfe663
I20260926 13:45:22.846058 20404 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:45:22.846252 20409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4651c1b3f0a1489b8b9d564073bfe663. Latest consensus state: current_term: 2 leader_uuid: "4651c1b3f0a1489b8b9d564073bfe663" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: false } } }
I20260926 13:45:22.846349 20409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:22.846586 20411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:45:22.848618 20411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:45:22.849017 20411 catalog_manager.cc:1259] Loaded cluster ID: d34e7363d79146f9bc88ca2ccf28bda6
I20260926 13:45:22.849059 20411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:45:22.849570 20411 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:45:22.849939 20411 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663: Loaded TSK: 0
I20260926 13:45:22.850566 20411 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:45:22.855082 19874 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:45:22.857913 19874 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:45:22.860316 20401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.860128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.0.0.1:49130)
0926 13:45:22.860225 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.860268 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 13:45:22.860405 19696 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [term 2 FOLLOWER]: Refusing update from remote peer 4651c1b3f0a1489b8b9d564073bfe663: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 13:45:22.860845 20409 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 13:45:22.860918 20401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.860706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.0.0.1:49134)
0926 13:45:22.860793 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.860846 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 13:45:22.861028 19863 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 -> Peer 0f4afcedc19e4050b7b7320081502c13 (127.12.170.126:33065): Couldn't send request to peer 0f4afcedc19e4050b7b7320081502c13. Status: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 13:45:22.862023 20404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4651c1b3f0a1489b8b9d564073bfe663. Latest consensus state: current_term: 2 leader_uuid: "4651c1b3f0a1489b8b9d564073bfe663" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: false } } }
I20260926 13:45:22.862129 20404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:45:22.862993 20399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "4651c1b3f0a1489b8b9d564073bfe663" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: false } } }
I20260926 13:45:22.863095 20399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663 [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:22.863482 20404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "4651c1b3f0a1489b8b9d564073bfe663" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "0f4afcedc19e4050b7b7320081502c13" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33065 } } peers { permanent_uuid: "8189bfdb296f41d69b7e564da513093f" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 40913 } attrs { promote: false } } peers { permanent_uuid: "4651c1b3f0a1489b8b9d564073bfe663" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 39363 } attrs { promote: false } } }
I20260926 13:45:22.863582 20404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8189bfdb296f41d69b7e564da513093f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:45:22.872092 20401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.868079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:33411 (local address 127.0.0.1:53684)
0926 13:45:22.868209 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.868223 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:22.868292 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:22.868780 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:22.868783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:22.868800 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:22.868894 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:22.868900 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.869768 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.869771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:22.870859 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:22.870867 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.871695 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.871698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:22.871804 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:22.871838 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 13:45:22.871917 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 13:45:22.871982 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":16}
I20260926 13:45:22.872109 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.868159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53684 (local address 127.12.170.65:33411)
0926 13:45:22.868570 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:22.868574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:22.868588 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:45:22.868637 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:22.868639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:22.868688 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:22.868774 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:22.868991 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.869688 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.870962 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.871554 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.871748 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:22.871813 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:22.871927 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 13:45:22.871985 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":63,"threads_started":1}
I20260926 13:45:22.872650 19989 tablet_service.cc:1467] Processing CreateTablet for tablet 4095d57dce8a4796af2387848c6e46ae (DEFAULT_TABLE table=test-table [id=c2185382822d422489d6c2170ded5a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:22.872900 19989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4095d57dce8a4796af2387848c6e46ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:22.873648 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.868660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:34977 (local address 127.0.0.1:48774)
0926 13:45:22.869278 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.869290 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:22.869399 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:22.869651 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:22.869653 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:22.869667 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:22.869753 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:22.869759 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.870720 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.870723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:22.872471 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:22.872481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.873305 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.873308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:22.873391 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:22.873413 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 13:45:22.873478 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 13:45:22.873534 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":3,"thread_start_us":68,"threads_started":1}
I20260926 13:45:22.873689 20421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.869079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48774 (local address 127.12.170.67:34977)
0926 13:45:22.869279 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:22.869284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:22.869386 (+ 102us) server_negotiation.cc:407] Connection header received
0926 13:45:22.869476 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:22.869479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:22.869527 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:22.869607 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:22.869842 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.870617 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.872589 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.873185 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.873272 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:22.873340 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:22.873497 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 13:45:22.873565 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":55,"threads_started":1}
I20260926 13:45:22.874058 20400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.868291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:33105 (local address 127.0.0.1:38982)
0926 13:45:22.868405 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.868417 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:22.868484 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:22.870967 (+ 2483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:22.870969 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:22.870987 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:22.871085 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:22.871090 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.872034 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.872037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:22.872970 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:22.872979 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.873787 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.873791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:22.873886 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:22.873903 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 13:45:22.873940 (+ 37us) client_negotiation.cc:240] Negotiation successful
0926 13:45:22.873985 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 13:45:22.874590 20420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.868882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38982 (local address 127.12.170.66:33105)
0926 13:45:22.869568 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:22.869572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:22.869584 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:45:22.869633 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:22.869636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:22.869681 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:22.869762 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:22.871201 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.871941 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.873103 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.873662 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.874320 (+ 658us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:22.874370 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:22.874398 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 13:45:22.874471 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":67,"threads_started":1}
I20260926 13:45:22.875223 20422 tablet_bootstrap.cc:492] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33: Bootstrap starting.
I20260926 13:45:22.875701 20126 tablet_service.cc:1467] Processing CreateTablet for tablet 4095d57dce8a4796af2387848c6e46ae (DEFAULT_TABLE table=test-table [id=c2185382822d422489d6c2170ded5a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:22.875865 20262 tablet_service.cc:1467] Processing CreateTablet for tablet 4095d57dce8a4796af2387848c6e46ae (DEFAULT_TABLE table=test-table [id=c2185382822d422489d6c2170ded5a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:22.875952 20126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4095d57dce8a4796af2387848c6e46ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:22.876084 20422 tablet_bootstrap.cc:654] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:22.876092 20262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4095d57dce8a4796af2387848c6e46ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:22.877238 20422 tablet_bootstrap.cc:492] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33: No bootstrap required, opened a new log
I20260926 13:45:22.877312 20422 ts_tablet_manager.cc:1397] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:22.877713 20422 raft_consensus.cc:348] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144cccc67d584072b304beeb25acad33" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 33411 } } peers { permanent_uuid: "5378a400ff1e490fb8c805c8465a4f37" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 33105 } } peers { permanent_uuid: "875eb6152ee6435683b49d64a8986403" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34977 } }
I20260926 13:45:22.877820 20422 raft_consensus.cc:374] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:22.877885 20422 raft_consensus.cc:729] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 144cccc67d584072b304beeb25acad33, State: Initialized, Role: FOLLOWER
I20260926 13:45:22.878001 20424 tablet_bootstrap.cc:492] T 4095d57dce8a4796af2387848c6e46ae P 5378a400ff1e490fb8c805c8465a4f37: Bootstrap starting.
I20260926 13:45:22.878038 20422 consensus_queue.cc:260] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144cccc67d584072b304beeb25acad33" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 33411 } } peers { permanent_uuid: "5378a400ff1e490fb8c805c8465a4f37" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 33105 } } peers { permanent_uuid: "875eb6152ee6435683b49d64a8986403" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34977 } }
I20260926 13:45:22.878978 20422 ts_tablet_manager.cc:1428] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:22.879112 20424 tablet_bootstrap.cc:654] T 4095d57dce8a4796af2387848c6e46ae P 5378a400ff1e490fb8c805c8465a4f37: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:22.880246 20424 tablet_bootstrap.cc:492] T 4095d57dce8a4796af2387848c6e46ae P 5378a400ff1e490fb8c805c8465a4f37: No bootstrap required, opened a new log
I20260926 13:45:22.880343 20424 ts_tablet_manager.cc:1397] T 4095d57dce8a4796af2387848c6e46ae P 5378a400ff1e490fb8c805c8465a4f37: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:22.880702 20424 raft_consensus.cc:348] T 4095d57dce8a4796af2387848c6e46ae P 5378a400ff1e490fb8c805c8465a4f37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144cccc67d584072b304beeb25acad33" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 33411 } } peers { permanent_uuid: "5378a400ff1e490fb8c805c8465a4f37" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 33105 } } peers { permanent_uuid: "875eb6152ee6435683b49d64a8986403" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34977 } }
I20260926 13:45:22.880807 20424 raft_consensus.cc:374] T 4095d57dce8a4796af2387848c6e46ae P 5378a400ff1e490fb8c805c8465a4f37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:22.880861 20055 heartbeater.cc:503] Master 127.12.170.124:39363 was elected leader, sending a full tablet report...
I20260926 13:45:22.880879 20424 raft_consensus.cc:729] T 4095d57dce8a4796af2387848c6e46ae P 5378a400ff1e490fb8c805c8465a4f37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5378a400ff1e490fb8c805c8465a4f37, State: Initialized, Role: FOLLOWER
I20260926 13:45:22.881016 20424 consensus_queue.cc:260] T 4095d57dce8a4796af2387848c6e46ae P 5378a400ff1e490fb8c805c8465a4f37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144cccc67d584072b304beeb25acad33" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 33411 } } peers { permanent_uuid: "5378a400ff1e490fb8c805c8465a4f37" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 33105 } } peers { permanent_uuid: "875eb6152ee6435683b49d64a8986403" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34977 } }
I20260926 13:45:22.881297 20424 ts_tablet_manager.cc:1428] T 4095d57dce8a4796af2387848c6e46ae P 5378a400ff1e490fb8c805c8465a4f37: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:45:22.879841 20427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.878935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.12.170.65:37767)
0926 13:45:22.879683 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.879737 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":82,"threads_started":1}
I20260926 13:45:22.882201 20430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.881852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.12.170.66:36209)
0926 13:45:22.882063 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.882116 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":85,"threads_started":1}
I20260926 13:45:22.882959 20431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.882666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.12.170.65:46789)
0926 13:45:22.882850 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.882892 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":71,"threads_started":1}
I20260926 13:45:22.882958 20425 tablet_bootstrap.cc:492] T 4095d57dce8a4796af2387848c6e46ae P 875eb6152ee6435683b49d64a8986403: Bootstrap starting.
I20260926 13:45:22.883831 20192 heartbeater.cc:503] Master 127.12.170.124:39363 was elected leader, sending a full tablet report...
I20260926 13:45:22.884142 20425 tablet_bootstrap.cc:654] T 4095d57dce8a4796af2387848c6e46ae P 875eb6152ee6435683b49d64a8986403: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:22.884819 20430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.884016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.12.170.66:59911)
0926 13:45:22.884711 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.884759 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":630}
I20260926 13:45:22.885269 20425 tablet_bootstrap.cc:492] T 4095d57dce8a4796af2387848c6e46ae P 875eb6152ee6435683b49d64a8986403: No bootstrap required, opened a new log
I20260926 13:45:22.885339 20425 ts_tablet_manager.cc:1397] T 4095d57dce8a4796af2387848c6e46ae P 875eb6152ee6435683b49d64a8986403: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:22.885628 20425 raft_consensus.cc:348] T 4095d57dce8a4796af2387848c6e46ae P 875eb6152ee6435683b49d64a8986403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144cccc67d584072b304beeb25acad33" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 33411 } } peers { permanent_uuid: "5378a400ff1e490fb8c805c8465a4f37" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 33105 } } peers { permanent_uuid: "875eb6152ee6435683b49d64a8986403" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34977 } }
I20260926 13:45:22.885710 20425 raft_consensus.cc:374] T 4095d57dce8a4796af2387848c6e46ae P 875eb6152ee6435683b49d64a8986403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:22.885733 20425 raft_consensus.cc:729] T 4095d57dce8a4796af2387848c6e46ae P 875eb6152ee6435683b49d64a8986403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 875eb6152ee6435683b49d64a8986403, State: Initialized, Role: FOLLOWER
I20260926 13:45:22.885861 20425 consensus_queue.cc:260] T 4095d57dce8a4796af2387848c6e46ae P 875eb6152ee6435683b49d64a8986403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144cccc67d584072b304beeb25acad33" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 33411 } } peers { permanent_uuid: "5378a400ff1e490fb8c805c8465a4f37" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 33105 } } peers { permanent_uuid: "875eb6152ee6435683b49d64a8986403" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34977 } }
I20260926 13:45:22.886149 20425 ts_tablet_manager.cc:1428] T 4095d57dce8a4796af2387848c6e46ae P 875eb6152ee6435683b49d64a8986403: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:45:22.886721 20434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.886372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.12.170.67:38767)
0926 13:45:22.886582 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.886636 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":83,"threads_started":1}
I20260926 13:45:22.887262 20434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.887075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.12.170.67:45435)
0926 13:45:22.887160 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.887206 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 13:45:22.887331 20328 heartbeater.cc:503] Master 127.12.170.124:39363 was elected leader, sending a full tablet report...
W20260926 13:45:22.889321 20195 tablet.cc:2369] T 4095d57dce8a4796af2387848c6e46ae P 5378a400ff1e490fb8c805c8465a4f37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:45:22.911885 20426 raft_consensus.cc:482] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:45:22.912007 20426 raft_consensus.cc:504] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144cccc67d584072b304beeb25acad33" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 33411 } } peers { permanent_uuid: "5378a400ff1e490fb8c805c8465a4f37" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 33105 } } peers { permanent_uuid: "875eb6152ee6435683b49d64a8986403" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34977 } }
I20260926 13:45:22.913002 20426 leader_election.cc:288] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5378a400ff1e490fb8c805c8465a4f37 (127.12.170.66:33105), 875eb6152ee6435683b49d64a8986403 (127.12.170.67:34977)
I20260926 13:45:22.917219 20420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.913264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:52999 (local address 127.12.170.66:33105)
0926 13:45:22.913373 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:22.913377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:22.913555 (+ 178us) server_negotiation.cc:407] Connection header received
0926 13:45:22.913643 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:22.913646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:22.913686 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:22.913746 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:22.914016 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.914621 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.916374 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.916749 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.916841 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:22.916880 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:22.917056 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 13:45:22.917128 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 13:45:22.917219 20431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.913178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:33105 (local address 127.12.170.65:52999)
0926 13:45:22.913441 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.913455 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:22.913555 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:22.913764 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:22.913767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:22.913791 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:22.913916 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:22.913921 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.914759 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.914762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:22.916254 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:22.916264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.916858 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.916860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:22.916940 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:22.916959 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 13:45:22.917021 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 13:45:22.917086 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"mutex_wait_us":101,"spinlock_wait_cycles":9856}
I20260926 13:45:22.917764 20137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4095d57dce8a4796af2387848c6e46ae" candidate_uuid: "144cccc67d584072b304beeb25acad33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5378a400ff1e490fb8c805c8465a4f37" is_pre_election: true
I20260926 13:45:22.917898 20427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.913222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:34977 (local address 127.12.170.65:37213)
0926 13:45:22.913442 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.913454 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:22.913546 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:22.913767 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:22.913770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:22.913793 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:22.913939 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:22.913944 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.914758 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.914762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:22.916254 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:22.916264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.917515 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.917519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:22.917619 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:22.917642 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 13:45:22.917713 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 13:45:22.917775 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":138,"spinlock_wait_cycles":8960}
I20260926 13:45:22.917901 20421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.913298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:37213 (local address 127.12.170.67:34977)
0926 13:45:22.913417 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:22.913421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:22.913523 (+ 102us) server_negotiation.cc:407] Connection header received
0926 13:45:22.913651 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:22.913654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:22.913691 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:22.913751 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:22.914044 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.914667 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.916923 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.917292 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.917421 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:22.917461 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:22.917733 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 13:45:22.917794 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:45:22.918061 20137 raft_consensus.cc:2454] T 4095d57dce8a4796af2387848c6e46ae P 5378a400ff1e490fb8c805c8465a4f37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 144cccc67d584072b304beeb25acad33 in term 0.
I20260926 13:45:22.918335 20282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4095d57dce8a4796af2387848c6e46ae" candidate_uuid: "144cccc67d584072b304beeb25acad33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875eb6152ee6435683b49d64a8986403" is_pre_election: true
I20260926 13:45:22.918466 20282 raft_consensus.cc:2454] T 4095d57dce8a4796af2387848c6e46ae P 875eb6152ee6435683b49d64a8986403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 144cccc67d584072b304beeb25acad33 in term 0.
I20260926 13:45:22.918483 19943 leader_election.cc:302] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 144cccc67d584072b304beeb25acad33, 5378a400ff1e490fb8c805c8465a4f37; no voters:
I20260926 13:45:22.918717 20426 raft_consensus.cc:2781] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:45:22.918772 20426 raft_consensus.cc:482] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:45:22.918798 20426 raft_consensus.cc:3037] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:22.919826 20426 raft_consensus.cc:504] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144cccc67d584072b304beeb25acad33" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 33411 } } peers { permanent_uuid: "5378a400ff1e490fb8c805c8465a4f37" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 33105 } } peers { permanent_uuid: "875eb6152ee6435683b49d64a8986403" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34977 } }
I20260926 13:45:22.920161 20426 leader_election.cc:288] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33 [CANDIDATE]: Term 1 election: Requested vote from peers 5378a400ff1e490fb8c805c8465a4f37 (127.12.170.66:33105), 875eb6152ee6435683b49d64a8986403 (127.12.170.67:34977)
I20260926 13:45:22.920425 20137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4095d57dce8a4796af2387848c6e46ae" candidate_uuid: "144cccc67d584072b304beeb25acad33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5378a400ff1e490fb8c805c8465a4f37"
I20260926 13:45:22.920439 20282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4095d57dce8a4796af2387848c6e46ae" candidate_uuid: "144cccc67d584072b304beeb25acad33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "875eb6152ee6435683b49d64a8986403"
I20260926 13:45:22.920522 20282 raft_consensus.cc:3037] T 4095d57dce8a4796af2387848c6e46ae P 875eb6152ee6435683b49d64a8986403 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:22.920527 20137 raft_consensus.cc:3037] T 4095d57dce8a4796af2387848c6e46ae P 5378a400ff1e490fb8c805c8465a4f37 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:22.921341 20282 raft_consensus.cc:2454] T 4095d57dce8a4796af2387848c6e46ae P 875eb6152ee6435683b49d64a8986403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 144cccc67d584072b304beeb25acad33 in term 1.
I20260926 13:45:22.921476 20137 raft_consensus.cc:2454] T 4095d57dce8a4796af2387848c6e46ae P 5378a400ff1e490fb8c805c8465a4f37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 144cccc67d584072b304beeb25acad33 in term 1.
I20260926 13:45:22.921634 19945 leader_election.cc:302] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 144cccc67d584072b304beeb25acad33, 875eb6152ee6435683b49d64a8986403; no voters:
I20260926 13:45:22.921803 20426 raft_consensus.cc:2781] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:22.921998 20426 raft_consensus.cc:686] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33 [term 1 LEADER]: Becoming Leader. State: Replica: 144cccc67d584072b304beeb25acad33, State: Running, Role: LEADER
I20260926 13:45:22.922216 20426 consensus_queue.cc:237] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144cccc67d584072b304beeb25acad33" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 33411 } } peers { permanent_uuid: "5378a400ff1e490fb8c805c8465a4f37" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 33105 } } peers { permanent_uuid: "875eb6152ee6435683b49d64a8986403" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34977 } }
I20260926 13:45:22.922559 20427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.922376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.12.170.65:35771)
0926 13:45:22.922463 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.922510 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 13:45:22.924445 19875 catalog_manager.cc:5495] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33 reported cstate change: term changed from 0 to 1, leader changed from <none> to 144cccc67d584072b304beeb25acad33 (127.12.170.65). New cstate: current_term: 1 leader_uuid: "144cccc67d584072b304beeb25acad33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "144cccc67d584072b304beeb25acad33" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 33411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5378a400ff1e490fb8c805c8465a4f37" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 33105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "875eb6152ee6435683b49d64a8986403" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34977 } health_report { overall_health: UNKNOWN } } }
I20260926 13:45:22.950079 20418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.946750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53694 (local address 127.12.170.65:33411)
0926 13:45:22.946873 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:22.946878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:22.946906 (+ 28us) server_negotiation.cc:407] Connection header received
0926 13:45:22.946981 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:22.946984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:22.947033 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:22.947117 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:45:22.947343 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.947923 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.948716 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:22.948979 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:22.949063 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:22.949310 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:45:22.949725 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:45:22.949826 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:22.949905 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 13:45:22.949954 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 13:45:22.952445 20282 raft_consensus.cc:1261] T 4095d57dce8a4796af2387848c6e46ae P 875eb6152ee6435683b49d64a8986403 [term 1 FOLLOWER]: Refusing update from remote peer 144cccc67d584072b304beeb25acad33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:22.952699 20137 raft_consensus.cc:1261] T 4095d57dce8a4796af2387848c6e46ae P 5378a400ff1e490fb8c805c8465a4f37 [term 1 FOLLOWER]: Refusing update from remote peer 144cccc67d584072b304beeb25acad33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:22.952967 20435 consensus_queue.cc:1035] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "875eb6152ee6435683b49d64a8986403" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:22.953202 20426 consensus_queue.cc:1035] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5378a400ff1e490fb8c805c8465a4f37" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 33105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:22.955482 20434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.954978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.12.170.67:56723)
0926 13:45:22.955359 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.955413 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303}
I20260926 13:45:22.956110 20430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.955206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.12.170.66:39829)
0926 13:45:22.955986 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.956043 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":712}
I20260926 13:45:22.957439 20427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.957241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.12.170.65:56285)
0926 13:45:22.957329 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.957377 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 13:45:22.957430 20440 mvcc.cc:205] Tried to move back new op lower bound from 7333602602808938496 to 7333602602691792896. Current Snapshot: MvccSnapshot[applied={T|T < 7333602602808938496}]
I20260926 13:45:22.958175 20427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.957942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.12.170.65:40065)
0926 13:45:22.958066 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.958115 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 13:45:22.958976 20434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.958762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.12.170.67:38333)
0926 13:45:22.958867 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.958912 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 13:45:22.959467 20430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.958797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33065 (local address 127.12.170.66:40689)
0926 13:45:22.959354 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.959407 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494}
I20260926 13:45:22.967348 19874 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:45:22.967476 19874 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:45:22.970549 19874 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4651c1b3f0a1489b8b9d564073bfe663: Sending DeleteTablet for 3 replicas of tablet 4095d57dce8a4796af2387848c6e46ae
I20260926 13:45:22.971131 19989 tablet_service.cc:1514] Processing DeleteTablet for tablet 4095d57dce8a4796af2387848c6e46ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:22 UTC) from {username='oryx', principal='oryx/127.12.170.124@KRBTEST.COM'} at 127.0.0.1:53684
I20260926 13:45:22.971411 20262 tablet_service.cc:1514] Processing DeleteTablet for tablet 4095d57dce8a4796af2387848c6e46ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:22 UTC) from {username='oryx', principal='oryx/127.12.170.124@KRBTEST.COM'} at 127.0.0.1:48774
I20260926 13:45:22.971527 20126 tablet_service.cc:1514] Processing DeleteTablet for tablet 4095d57dce8a4796af2387848c6e46ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:22 UTC) from {username='oryx', principal='oryx/127.12.170.124@KRBTEST.COM'} at 127.0.0.1:38982
I20260926 13:45:22.971755 20448 tablet_replica.cc:330] T 4095d57dce8a4796af2387848c6e46ae P 875eb6152ee6435683b49d64a8986403: stopping tablet replica
I20260926 13:45:22.971903 20448 raft_consensus.cc:2229] T 4095d57dce8a4796af2387848c6e46ae P 875eb6152ee6435683b49d64a8986403 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:45:22.972074 20447 tablet_replica.cc:330] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33: stopping tablet replica
I20260926 13:45:22.972186 20447 raft_consensus.cc:2229] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:45:22.972491 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 19830
I20260926 13:45:22.972493 20447 raft_consensus.cc:2258] T 4095d57dce8a4796af2387848c6e46ae P 144cccc67d584072b304beeb25acad33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:22.973026 20449 tablet_replica.cc:330] T 4095d57dce8a4796af2387848c6e46ae P 5378a400ff1e490fb8c805c8465a4f37: stopping tablet replica
I20260926 13:45:22.973129 20449 raft_consensus.cc:2229] T 4095d57dce8a4796af2387848c6e46ae P 5378a400ff1e490fb8c805c8465a4f37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:45:22.973229 20449 raft_consensus.cc:2258] T 4095d57dce8a4796af2387848c6e46ae P 5378a400ff1e490fb8c805c8465a4f37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:22.974133 20449 ts_tablet_manager.cc:1905] T 4095d57dce8a4796af2387848c6e46ae P 5378a400ff1e490fb8c805c8465a4f37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:22.974347 20448 raft_consensus.cc:2258] T 4095d57dce8a4796af2387848c6e46ae P 875eb6152ee6435683b49d64a8986403 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:22.976127 20448 ts_tablet_manager.cc:1905] T 4095d57dce8a4796af2387848c6e46ae P 875eb6152ee6435683b49d64a8986403: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:22.976693 20449 ts_tablet_manager.cc:1918] T 4095d57dce8a4796af2387848c6e46ae P 5378a400ff1e490fb8c805c8465a4f37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:45:22.976768 20449 log.cc:1196] T 4095d57dce8a4796af2387848c6e46ae P 5378a400ff1e490fb8c805c8465a4f37: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-1/wal/wals/4095d57dce8a4796af2387848c6e46ae
I20260926 13:45:22.977022 20449 ts_tablet_manager.cc:1939] T 4095d57dce8a4796af2387848c6e46ae P 5378a400ff1e490fb8c805c8465a4f37: Deleting consensus metadata
I20260926 13:45:22.977653 19861 catalog_manager.cc:4841] TS 5378a400ff1e490fb8c805c8465a4f37 (127.12.170.66:33105): tablet 4095d57dce8a4796af2387848c6e46ae (table test-table [id=c2185382822d422489d6c2170ded5a94]) successfully deleted
I20260926 13:45:22.978055 20448 ts_tablet_manager.cc:1918] T 4095d57dce8a4796af2387848c6e46ae P 875eb6152ee6435683b49d64a8986403: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:45:22.978112 20448 log.cc:1196] T 4095d57dce8a4796af2387848c6e46ae P 875eb6152ee6435683b49d64a8986403: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/4095d57dce8a4796af2387848c6e46ae
I20260926 13:45:22.978266 20448 ts_tablet_manager.cc:1939] T 4095d57dce8a4796af2387848c6e46ae P 875eb6152ee6435683b49d64a8986403: Deleting consensus metadata
I20260926 13:45:22.978935 19863 catalog_manager.cc:4841] TS 875eb6152ee6435683b49d64a8986403 (127.12.170.67:34977): tablet 4095d57dce8a4796af2387848c6e46ae (table test-table [id=c2185382822d422489d6c2170ded5a94]) successfully deleted
I20260926 13:45:22.980037 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 20063
I20260926 13:45:22.980237 20400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:22.979991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:33411 (local address 127.0.0.1:53702)
0926 13:45:22.980112 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:22.980165 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.65:33411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 13:45:22.981667 19860 catalog_manager.cc:4584] TS 144cccc67d584072b304beeb25acad33 (127.12.170.65:33411): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4095d57dce8a4796af2387848c6e46ae: Network error: Client connection negotiation failed: client connection to 127.12.170.65:33411: connect: Connection refused (error 111)
I20260926 13:45:22.985690 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 20199
I20260926 13:45:22.990310 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 19543
I20260926 13:45:22.994707 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 19651
2026-09-26T13:45:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4949 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:45:23 dist-test-slave-hc1k krb5kdc[20452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:45:23 dist-test-slave-hc1k krb5kdc[20452](info): set up 2 sockets
Sep 26 13:45:23 dist-test-slave-hc1k krb5kdc[20452](info): commencing operation
krb5kdc: starting...
W20260926 13:45:23.947995 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:45:24 dist-test-slave-hc1k krb5kdc[20452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:45:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:45:24Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.126.keytab.
Entry for principal oryx/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.126.keytab.
I20260926 13:45:24.054174 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:43289
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.126.keytab
--principal=oryx/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:34347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:43289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:24.156427 20465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:24.156657 20465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:24.156706 20465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:24.161093 20465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:45:24.161155 20465 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 13:45:24.161180 20465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:24.161204 20465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:24.161224 20465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:45:24.161257 20465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:45:24.166500 20465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:34347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:43289
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.126.keytab
--principal=oryx/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:43289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:24.167408 20465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:24.168360 20465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:24.176182 20471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:24.176167 20473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:24.176403 20465 server_base.cc:1034] running on GCE node
W20260926 13:45:24.176411 20470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:24.176954 20465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:24.177932 20465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:24.179086 20465 hybrid_clock.cc:648] HybridClock initialized: now 1790430324179056 us; error 46 us; skew 500 ppm
Sep 26 13:45:24 dist-test-slave-hc1k krb5kdc[20452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430324, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:24.181715 20465 init.cc:377] Logged in from keytab as oryx/127.12.170.126@KRBTEST.COM (short username oryx)
I20260926 13:45:24.182538 20465 webserver.cc:460] Webserver started at http://127.12.170.126:39687/ using document root <none> and password file <none>
I20260926 13:45:24.183066 20465 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:24.183131 20465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:24.183303 20465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:24.185204 20465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "5b7c2d92ec29421aad8eee9cfe59c98e"
format_stamp: "Formatted at 2026-09-26 13:45:24 on dist-test-slave-hc1k"
server_key: "bfd690df7f9b99f5a25aaf13baaf7a23"
server_key_iv: "a052ad3d6c057e5c5d25c593e739a8a4"
server_key_version: "encryptionkey@0"
I20260926 13:45:24.185711 20465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "5b7c2d92ec29421aad8eee9cfe59c98e"
format_stamp: "Formatted at 2026-09-26 13:45:24 on dist-test-slave-hc1k"
server_key: "bfd690df7f9b99f5a25aaf13baaf7a23"
server_key_iv: "a052ad3d6c057e5c5d25c593e739a8a4"
server_key_version: "encryptionkey@0"
I20260926 13:45:24.189100 20465 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 13:45:24.191322 20480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:24.192489 20465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:24.192602 20465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "5b7c2d92ec29421aad8eee9cfe59c98e"
format_stamp: "Formatted at 2026-09-26 13:45:24 on dist-test-slave-hc1k"
server_key: "bfd690df7f9b99f5a25aaf13baaf7a23"
server_key_iv: "a052ad3d6c057e5c5d25c593e739a8a4"
server_key_version: "encryptionkey@0"
I20260926 13:45:24.192690 20465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:24.210561 20465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:24.213125 20465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:24.213276 20465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:24.220498 20465 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:43289
I20260926 13:45:24.220511 20542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:43289 every 8 connection(s)
I20260926 13:45:24.221346 20465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:45:24.223996 20543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:24.228961 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 20465
I20260926 13:45:24.229080 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:45:24.229202 20543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e: Bootstrap starting.
I20260926 13:45:24.229380 12969 external_mini_cluster.cc:1428] Setting key 95fcbaf555b1b3df8870853990855009
I20260926 13:45:24.231534 20543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:24.232225 20543 log.cc:824] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:24.234005 20543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e: No bootstrap required, opened a new log
Sep 26 13:45:24 dist-test-slave-hc1k krb5kdc[20452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM
I20260926 13:45:24.237967 20543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7c2d92ec29421aad8eee9cfe59c98e" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 43289 } }
I20260926 13:45:24.238255 20543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:24.238319 20543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7c2d92ec29421aad8eee9cfe59c98e, State: Initialized, Role: FOLLOWER
I20260926 13:45:24.238731 20543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7c2d92ec29421aad8eee9cfe59c98e" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 43289 } }
I20260926 13:45:24.238893 20543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:45:24.238960 20543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:45:24.239048 20543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:24.239894 20543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7c2d92ec29421aad8eee9cfe59c98e" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 43289 } }
I20260926 13:45:24.240201 20543 leader_election.cc:302] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b7c2d92ec29421aad8eee9cfe59c98e; no voters:
I20260926 13:45:24.240523 20543 leader_election.cc:288] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:45:24.240602 20548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:24.240792 20548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e [term 1 LEADER]: Becoming Leader. State: Replica: 5b7c2d92ec29421aad8eee9cfe59c98e, State: Running, Role: LEADER
I20260926 13:45:24.241082 20548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7c2d92ec29421aad8eee9cfe59c98e" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 43289 } }
I20260926 13:45:24.241680 20543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:45:24.242776 20550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b7c2d92ec29421aad8eee9cfe59c98e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7c2d92ec29421aad8eee9cfe59c98e" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 43289 } } }
I20260926 13:45:24.242888 20550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:24.242995 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:24.230548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36606 (local address 127.12.170.126:43289)
0926 13:45:24.230965 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:24.230980 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:24.231014 (+ 34us) server_negotiation.cc:407] Connection header received
0926 13:45:24.231701 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:24.231720 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:24.232029 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:24.232287 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:24.233060 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.234001 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.235104 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.235552 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.235692 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:24.238247 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:24.238253 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:24.238269 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:24.238301 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:24.240066 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:24.240519 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:24.240525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:24.240531 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:24.240669 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:24.240982 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:24.240985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:24.240987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:24.241458 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:24.241654 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:24.241897 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 13:45:24.242170 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":117,"threads_started":1}
I20260926 13:45:24.243228 20557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:45:24.243491 20549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b7c2d92ec29421aad8eee9cfe59c98e. Latest consensus state: current_term: 1 leader_uuid: "5b7c2d92ec29421aad8eee9cfe59c98e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b7c2d92ec29421aad8eee9cfe59c98e" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 43289 } } }
I20260926 13:45:24.243701 20549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:24.246176 20557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:45:24.250967 20557 catalog_manager.cc:1347] Generated new cluster ID: 783599c95f9d4471bcf7ffbcead53d76
I20260926 13:45:24.251022 20557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:45:24.265849 20557 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:45:24.266664 20557 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:45:24.275377 20557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e: Generated new TSK 0
I20260926 13:45:24.276046 20557 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.170.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.170.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.170.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.65.keytab.
Entry for principal oryx/127.12.170.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.65.keytab.
I20260926 13:45:24.310324 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:0
--local_ip_for_outbound_sockets=127.12.170.65
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:43289
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.65.keytab
--principal=oryx/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:34347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:24.415737 20569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:24.415971 20569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:24.416015 20569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:24.419803 20569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:24.419845 20569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:24.419880 20569 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:45:24.419915 20569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:45:24.424830 20569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:34347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.65.keytab
--principal=oryx/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.170.126:43289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:24.425649 20569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:24.426513 20569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:24.434880 20575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:24.434907 20577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:24.434902 20574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:24.434940 20569 server_base.cc:1034] running on GCE node
I20260926 13:45:24.435449 20569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:24.436025 20569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:24.437162 20569 hybrid_clock.cc:648] HybridClock initialized: now 1790430324437137 us; error 36 us; skew 500 ppm
Sep 26 13:45:24 dist-test-slave-hc1k krb5kdc[20452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430324, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:24.439477 20569 init.cc:377] Logged in from keytab as oryx/127.12.170.65@KRBTEST.COM (short username oryx)
I20260926 13:45:24.440243 20569 webserver.cc:460] Webserver started at http://127.12.170.65:40467/ using document root <none> and password file <none>
I20260926 13:45:24.440712 20569 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:24.440773 20569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:24.440937 20569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:24.442806 20569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-0/data/instance:
uuid: "0daec48270fb4665beb58e1cb1c672e3"
format_stamp: "Formatted at 2026-09-26 13:45:24 on dist-test-slave-hc1k"
server_key: "88f67e0b82b5439ccd129d6075552b2b"
server_key_iv: "7a24245900ddc87d7a7623eaaa60411e"
server_key_version: "encryptionkey@0"
I20260926 13:45:24.443305 20569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance:
uuid: "0daec48270fb4665beb58e1cb1c672e3"
format_stamp: "Formatted at 2026-09-26 13:45:24 on dist-test-slave-hc1k"
server_key: "88f67e0b82b5439ccd129d6075552b2b"
server_key_iv: "7a24245900ddc87d7a7623eaaa60411e"
server_key_version: "encryptionkey@0"
I20260926 13:45:24.446511 20569 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260926 13:45:24.448639 20584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:24.449693 20569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:24.449787 20569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "0daec48270fb4665beb58e1cb1c672e3"
format_stamp: "Formatted at 2026-09-26 13:45:24 on dist-test-slave-hc1k"
server_key: "88f67e0b82b5439ccd129d6075552b2b"
server_key_iv: "7a24245900ddc87d7a7623eaaa60411e"
server_key_version: "encryptionkey@0"
I20260926 13:45:24.449869 20569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:24.458599 20569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:24.461081 20569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:24.461226 20569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:24.462432 20569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:24.462478 20569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:24.462525 20569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:24.462553 20569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 13:45:24 dist-test-slave-hc1k krb5kdc[20452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430324, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM
I20260926 13:45:24.471683 20569 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:45189
I20260926 13:45:24.471783 20699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:45189 every 8 connection(s)
I20260926 13:45:24.472520 20569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:45:24.475170 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:24.463111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:56457 (local address 127.12.170.126:43289)
0926 13:45:24.463242 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:24.463247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:24.464061 (+ 814us) server_negotiation.cc:407] Connection header received
0926 13:45:24.465082 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:24.465086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:24.465137 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:24.465190 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:24.466653 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.467448 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.469076 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.469333 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.469412 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:24.472143 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:24.472146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:24.472148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:24.472170 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:24.473558 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:24.474023 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:24.474026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:24.474027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:24.474097 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:24.474342 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:24.474344 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:24.474345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:24.474496 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:24.474560 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:24.474920 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 13:45:24.475020 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 13:45:24.475257 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 20569
I20260926 13:45:24.475355 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance
I20260926 13:45:24.475607 12969 external_mini_cluster.cc:1428] Setting key a2dc5421a89f69b6e738b74a5f7f0101
I20260926 13:45:24.475880 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:24.463446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:43289 (local address 127.12.170.65:56457)
0926 13:45:24.463916 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:24.463962 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:24.464856 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:24.465557 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:24.465566 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:24.465940 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:24.466520 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:24.466540 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.467586 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.467590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:24.468952 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:24.468963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.469495 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.469500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:24.469692 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:24.470260 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:24.470321 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:24.471997 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:24.473654 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:24.473660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:24.473672 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:24.473938 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:24.474158 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:24.474160 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:24.474161 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:24.474278 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:24.474559 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:24.474563 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:24.474750 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 13:45:24.474882 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 13:45:24.475088 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":97,"threads_started":1}
I20260926 13:45:24.477119 20700 heartbeater.cc:347] Connected to a master server at 127.12.170.126:43289
I20260926 13:45:24.477247 20700 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:24.477751 20700 heartbeater.cc:511] Master 127.12.170.126:43289 requested a full tablet report, sending...
I20260926 13:45:24.479192 20497 ts_manager.cc:194] Registered new tserver with Master: 0daec48270fb4665beb58e1cb1c672e3 (127.12.170.65:45189)
I20260926 13:45:24.480279 20497 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.170.65@KRBTEST.COM'} at 127.12.170.65:56457
WARNING: no policy specified for oryx/127.12.170.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.170.66@KRBTEST.COM" created.
I20260926 13:45:24.487046 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:24.478970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:43289 (local address 127.12.170.65:41719)
0926 13:45:24.479256 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:24.479269 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:24.480924 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:24.481165 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:24.481168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:24.481362 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:24.481513 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:24.481521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.482448 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.482452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:24.483316 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:24.483326 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.483721 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.483724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:24.483822 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:24.484205 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:24.484218 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:24.484595 (+ 377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:24.485845 (+ 1250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:24.485848 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:24.485849 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:24.486018 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:24.486222 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:24.486225 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:24.486227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:24.486306 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:24.486600 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:24.486603 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:24.486679 (+ 76us) client_negotiation.cc:769] Sending connection context
0926 13:45:24.486767 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 13:45:24.486873 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":1,"thread_start_us":88,"threads_started":1}
I20260926 13:45:24.487160 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:24.479042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:41719 (local address 127.12.170.126:43289)
0926 13:45:24.479135 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:24.479139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:24.479328 (+ 189us) server_negotiation.cc:407] Connection header received
0926 13:45:24.481013 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:24.481016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:24.481059 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:24.481137 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:24.481627 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.482325 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.483415 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.483606 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.483702 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:24.484707 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:24.484710 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:24.484712 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:24.484731 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:24.485759 (+ 1028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:24.486090 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:24.486092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:24.486093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:24.486152 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:24.486385 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:24.486387 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:24.486389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:24.486532 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:24.486606 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:24.486916 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 13:45:24.487017 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.170.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.66.keytab.
Entry for principal oryx/127.12.170.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.66.keytab.
I20260926 13:45:24.499439 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:0
--local_ip_for_outbound_sockets=127.12.170.66
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:43289
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.66.keytab
--principal=oryx/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:34347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:24.602022 20709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:24.602272 20709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:24.602313 20709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:24.606278 20709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:24.606339 20709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:24.606377 20709 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:45:24.606415 20709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:45:24.611472 20709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:34347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.66.keytab
--principal=oryx/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.170.126:43289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:24.612460 20709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:24.613420 20709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:24.622479 20714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:24.622473 20717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:24.622478 20715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:24.622602 20709 server_base.cc:1034] running on GCE node
I20260926 13:45:24.623176 20709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:24.623754 20709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:24.624909 20709 hybrid_clock.cc:648] HybridClock initialized: now 1790430324624896 us; error 28 us; skew 500 ppm
Sep 26 13:45:24 dist-test-slave-hc1k krb5kdc[20452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430324, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:24.627472 20709 init.cc:377] Logged in from keytab as oryx/127.12.170.66@KRBTEST.COM (short username oryx)
I20260926 13:45:24.628330 20709 webserver.cc:460] Webserver started at http://127.12.170.66:41159/ using document root <none> and password file <none>
I20260926 13:45:24.628851 20709 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:24.628914 20709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:24.629093 20709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:24.631052 20709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-1/data/instance:
uuid: "556c77c7adf74f3b83543fea8130ed35"
format_stamp: "Formatted at 2026-09-26 13:45:24 on dist-test-slave-hc1k"
server_key: "e0d36a764aae003a192ed51b767becd8"
server_key_iv: "e26ee5ae233681c8f65dd6488b90c057"
server_key_version: "encryptionkey@0"
I20260926 13:45:24.631614 20709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance:
uuid: "556c77c7adf74f3b83543fea8130ed35"
format_stamp: "Formatted at 2026-09-26 13:45:24 on dist-test-slave-hc1k"
server_key: "e0d36a764aae003a192ed51b767becd8"
server_key_iv: "e26ee5ae233681c8f65dd6488b90c057"
server_key_version: "encryptionkey@0"
I20260926 13:45:24.635032 20709 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:45:24.637392 20724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:24.638535 20709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:24.638631 20709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "556c77c7adf74f3b83543fea8130ed35"
format_stamp: "Formatted at 2026-09-26 13:45:24 on dist-test-slave-hc1k"
server_key: "e0d36a764aae003a192ed51b767becd8"
server_key_iv: "e26ee5ae233681c8f65dd6488b90c057"
server_key_version: "encryptionkey@0"
I20260926 13:45:24.638717 20709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:24.652128 20709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:24.654987 20709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:24.655148 20709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:24.656551 20709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:24.656598 20709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:24.656649 20709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:24.656677 20709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 13:45:24 dist-test-slave-hc1k krb5kdc[20452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430324, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM
I20260926 13:45:24.666131 20709 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:42103
I20260926 13:45:24.666175 20839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:42103 every 8 connection(s)
I20260926 13:45:24.667059 20709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:45:24.669395 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:24.657175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:47259 (local address 127.12.170.126:43289)
0926 13:45:24.657309 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:24.657313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:24.658176 (+ 863us) server_negotiation.cc:407] Connection header received
0926 13:45:24.659165 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:24.659168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:24.659222 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:24.659314 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:24.660702 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.661516 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.663124 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.663453 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.663568 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:24.666362 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:24.666366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:24.666369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:24.666393 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:24.667764 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:24.668225 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:24.668228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:24.668229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:24.668301 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:24.668531 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:24.668533 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:24.668534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:24.668686 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:24.668785 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:24.669147 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 13:45:24.669239 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 13:45:24.670084 20733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:24.657547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:43289 (local address 127.12.170.66:47259)
0926 13:45:24.658010 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:24.658058 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:24.658946 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:24.659478 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:24.659487 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:24.659946 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:24.660506 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:24.660525 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.661646 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.661650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:24.663001 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:24.663013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.663676 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.663681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:24.663879 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:24.664464 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:24.664486 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:24.666167 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:24.667869 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:24.667874 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:24.667886 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:24.668150 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:24.668357 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:24.668359 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:24.668361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:24.668473 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:24.668753 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:24.668757 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:24.668973 (+ 216us) client_negotiation.cc:769] Sending connection context
0926 13:45:24.669115 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 13:45:24.669336 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":141,"threads_started":1}
I20260926 13:45:24.671101 20840 heartbeater.cc:347] Connected to a master server at 127.12.170.126:43289
I20260926 13:45:24.671226 20840 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:24.671702 20840 heartbeater.cc:511] Master 127.12.170.126:43289 requested a full tablet report, sending...
I20260926 13:45:24.672868 20495 ts_manager.cc:194] Registered new tserver with Master: 556c77c7adf74f3b83543fea8130ed35 (127.12.170.66:42103)
I20260926 13:45:24.673509 20495 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.170.66@KRBTEST.COM'} at 127.12.170.66:47259
I20260926 13:45:24.674490 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 20709
I20260926 13:45:24.674619 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance
I20260926 13:45:24.674952 12969 external_mini_cluster.cc:1428] Setting key caf9405c60842a103304ff315c51c6f2
I20260926 13:45:24.680536 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:24.673049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:50221 (local address 127.12.170.126:43289)
0926 13:45:24.673143 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:24.673146 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:24.674000 (+ 854us) server_negotiation.cc:407] Connection header received
0926 13:45:24.674123 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:24.674125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:24.674169 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:24.674260 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:24.674735 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.675447 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.676368 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.676572 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.676645 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:24.677740 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:24.677742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:24.677744 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:24.677763 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:24.679038 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:24.679453 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:24.679455 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:24.679456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:24.679518 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:24.679787 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:24.679788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:24.679789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:24.679983 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:24.680054 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:24.680346 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 13:45:24.680433 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 13:45:24.680644 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:24.672978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:43289 (local address 127.12.170.66:50221)
0926 13:45:24.673910 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:24.673927 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:24.674033 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:24.674296 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:24.674300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:24.674497 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:24.674631 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:24.674640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.675567 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.675571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:24.676260 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:24.676272 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.676676 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.676680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:24.676800 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:24.677215 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:24.677230 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:24.677622 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:24.679118 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:24.679122 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:24.679124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:24.679366 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:24.679590 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:24.679593 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:24.679595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:24.679709 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:24.680068 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:24.680072 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:24.680168 (+ 96us) client_negotiation.cc:769] Sending connection context
0926 13:45:24.680298 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 13:45:24.680432 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"thread_start_us":96,"threads_started":1}
WARNING: no policy specified for oryx/127.12.170.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.170.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.170.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.67.keytab.
Entry for principal oryx/127.12.170.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.67.keytab.
I20260926 13:45:24.699419 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:0
--local_ip_for_outbound_sockets=127.12.170.67
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:43289
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.67.keytab
--principal=oryx/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:34347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:24.801548 20849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:24.801762 20849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:24.801802 20849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:24.805680 20849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:24.805725 20849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:24.805763 20849 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:45:24.805814 20849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:45:24.810614 20849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:34347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.67.keytab
--principal=oryx/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.170.126:43289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:24.811465 20849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:24.812355 20849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:45:24.821084 20849 server_base.cc:1034] running on GCE node
W20260926 13:45:24.821036 20854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:24.821053 20855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:24.821053 20857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:24.821630 20849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:24.822139 20849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:24.823288 20849 hybrid_clock.cc:648] HybridClock initialized: now 1790430324823268 us; error 30 us; skew 500 ppm
Sep 26 13:45:24 dist-test-slave-hc1k krb5kdc[20452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430324, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:24.825779 20849 init.cc:377] Logged in from keytab as oryx/127.12.170.67@KRBTEST.COM (short username oryx)
I20260926 13:45:24.826519 20849 webserver.cc:460] Webserver started at http://127.12.170.67:36341/ using document root <none> and password file <none>
I20260926 13:45:24.826994 20849 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:24.827050 20849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:24.827214 20849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:24.829098 20849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-2/data/instance:
uuid: "ff63674a2a8848d1a168dfbd580bbf3a"
format_stamp: "Formatted at 2026-09-26 13:45:24 on dist-test-slave-hc1k"
server_key: "49e4f64d02eb348e9afc96033c26c3ae"
server_key_iv: "ff745c41e49b5342ece77c47bef8ca81"
server_key_version: "encryptionkey@0"
I20260926 13:45:24.829591 20849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance:
uuid: "ff63674a2a8848d1a168dfbd580bbf3a"
format_stamp: "Formatted at 2026-09-26 13:45:24 on dist-test-slave-hc1k"
server_key: "49e4f64d02eb348e9afc96033c26c3ae"
server_key_iv: "ff745c41e49b5342ece77c47bef8ca81"
server_key_version: "encryptionkey@0"
I20260926 13:45:24.832933 20849 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:45:24.835104 20864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:24.836349 20849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:24.836441 20849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "ff63674a2a8848d1a168dfbd580bbf3a"
format_stamp: "Formatted at 2026-09-26 13:45:24 on dist-test-slave-hc1k"
server_key: "49e4f64d02eb348e9afc96033c26c3ae"
server_key_iv: "ff745c41e49b5342ece77c47bef8ca81"
server_key_version: "encryptionkey@0"
I20260926 13:45:24.836531 20849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:24.849403 20849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:24.851872 20849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:24.852016 20849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:24.853374 20849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:24.853425 20849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:24.853475 20849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:24.853502 20849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 13:45:24 dist-test-slave-hc1k krb5kdc[20452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430324, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM
I20260926 13:45:24.863016 20849 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:44257
I20260926 13:45:24.863093 20979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:44257 every 8 connection(s)
I20260926 13:45:24.863849 20849 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:45:24.864053 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 20849
I20260926 13:45:24.864145 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance
I20260926 13:45:24.864434 12969 external_mini_cluster.cc:1428] Setting key 63cedc6728c11ea4b0d6bc29160ce984
I20260926 13:45:24.866565 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:24.853979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:42195 (local address 127.12.170.126:43289)
0926 13:45:24.854095 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:24.854098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:24.854962 (+ 864us) server_negotiation.cc:407] Connection header received
0926 13:45:24.855922 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:24.855926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:24.855984 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:24.856055 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:24.857532 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.858332 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.859997 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.860328 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.860426 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:24.862965 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:24.862969 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:24.862972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:24.863000 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:24.864609 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:24.865123 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:24.865127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:24.865129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:24.865217 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:24.865567 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:24.865570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:24.865572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:24.865768 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:24.865840 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:24.866362 (+ 522us) server_negotiation.cc:299] Negotiation successful
0926 13:45:24.866454 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 13:45:24.867276 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:24.854388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:43289 (local address 127.12.170.67:42195)
0926 13:45:24.854822 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:24.854859 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:24.855725 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:24.856393 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:24.856401 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:24.856824 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:24.857352 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:24.857381 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.858483 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.858487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:24.859876 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:24.859887 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.860476 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.860479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:24.860674 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:24.861157 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:24.861179 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:24.862808 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:24.864729 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:24.864735 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:24.864747 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:24.865028 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:24.865301 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:24.865304 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:24.865306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:24.865471 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:24.865856 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:24.865862 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:24.866124 (+ 262us) client_negotiation.cc:769] Sending connection context
0926 13:45:24.866318 (+ 194us) client_negotiation.cc:240] Negotiation successful
0926 13:45:24.866592 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":123,"threads_started":1}
I20260926 13:45:24.868211 20980 heartbeater.cc:347] Connected to a master server at 127.12.170.126:43289
I20260926 13:45:24.868331 20980 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:24.868815 20980 heartbeater.cc:511] Master 127.12.170.126:43289 requested a full tablet report, sending...
I20260926 13:45:24.869730 20497 ts_manager.cc:194] Registered new tserver with Master: ff63674a2a8848d1a168dfbd580bbf3a (127.12.170.67:44257)
I20260926 13:45:24.870388 20497 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.170.67@KRBTEST.COM'} at 127.12.170.67:42195
I20260926 13:45:24.875963 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:24.870192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:43289 (local address 127.12.170.67:34429)
0926 13:45:24.870476 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:24.870489 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:24.870573 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:24.870840 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:24.870843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:24.871001 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:24.871093 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:24.871098 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.871910 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.871913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:24.872488 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:24.872496 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.872861 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.872863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:24.872922 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:24.873319 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:24.873330 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:24.873597 (+ 267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:24.874872 (+ 1275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:24.874874 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:24.874876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:24.875045 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:24.875228 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:24.875230 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:24.875231 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:24.875293 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:24.875555 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:24.875557 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:24.875619 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 13:45:24.875741 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 13:45:24.875842 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":2,"thread_start_us":82,"threads_started":1}
I20260926 13:45:24.875967 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:24.870254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:34429 (local address 127.12.170.126:43289)
0926 13:45:24.870358 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:24.870362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:24.870540 (+ 178us) server_negotiation.cc:407] Connection header received
0926 13:45:24.870666 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:24.870668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:24.870710 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:24.870773 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:24.871205 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.871782 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.872580 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.872780 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.872856 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:24.873716 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:24.873718 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:24.873720 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:24.873740 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:24.874782 (+ 1042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:24.875111 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:24.875113 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:24.875115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:24.875173 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:24.875347 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:24.875349 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:24.875350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:24.875496 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:24.875560 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:24.875776 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 13:45:24.875852 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 13:45:24.877907 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 13:45:24 dist-test-slave-hc1k krb5kdc[20452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430324, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 13:45:24 dist-test-slave-hc1k krb5kdc[20452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:24 dist-test-slave-hc1k krb5kdc[20452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 13:45:24.898104 20546 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:24.892912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36614 (local address 127.12.170.126:43289)
0926 13:45:24.893046 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:24.893049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:24.893060 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:45:24.893091 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:24.893093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:24.893127 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:24.893186 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:24.893763 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.894244 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.895019 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.895207 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.895269 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:24.898005 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36614: BlockingRecv error: recv got EOF from 127.0.0.1:36614 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 13:45:24 dist-test-slave-hc1k krb5kdc[20452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430324, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM
I20260926 13:45:24.906353 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:24.899216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36616 (local address 127.12.170.126:43289)
0926 13:45:24.899326 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:24.899330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:24.899445 (+ 115us) server_negotiation.cc:407] Connection header received
0926 13:45:24.899555 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:24.899558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:24.899589 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:24.899645 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:24.900230 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.900684 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.901412 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.901620 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.901698 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:24.903612 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:24.903614 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:24.903616 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:24.903635 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:24.904783 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:24.905192 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:24.905196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:24.905198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:24.905291 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:24.905548 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:24.905552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:24.905554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:24.905725 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:24.905855 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:24.906083 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 13:45:24.906184 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 13:45:24.908762 20497 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:45:24.911696 20497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:45:24.925092 21002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:24.919268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:44257 (local address 127.0.0.1:60918)
0926 13:45:24.919846 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:24.919894 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:24.920428 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:24.920958 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:24.920965 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:24.920988 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:24.921112 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:24.921120 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.922241 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.922245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:24.923595 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:24.923605 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.924538 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.924542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:24.924657 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:24.924700 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 13:45:24.924803 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 13:45:24.924929 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":159,"thread_start_us":86,"threads_started":1}
I20260926 13:45:24.925216 21007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:24.920422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50642 (local address 127.12.170.65:45189)
0926 13:45:24.920725 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:24.920736 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:24.920757 (+ 21us) server_negotiation.cc:407] Connection header received
0926 13:45:24.920827 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:24.920832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:24.921022 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:24.921170 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:24.921584 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.922641 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.924163 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.924680 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.924773 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:24.924919 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:24.925006 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 13:45:24.925085 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":113,"threads_started":1}
I20260926 13:45:24.925477 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:24.919713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:42103 (local address 127.0.0.1:43108)
0926 13:45:24.920138 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:24.920151 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:24.920246 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:24.921184 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:24.921186 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:24.921202 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:24.921330 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:24.921337 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.922656 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.922659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:24.924064 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:24.924071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.925091 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.925096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:24.925188 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:24.925215 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 13:45:24.925282 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 13:45:24.925348 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":233,"thread_start_us":79,"threads_started":1}
I20260926 13:45:24.925589 21005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:24.920015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:45189 (local address 127.0.0.1:50642)
0926 13:45:24.920504 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:24.920515 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:24.920584 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:24.921141 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:24.921144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:24.921160 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:24.921465 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:24.921472 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.922742 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.922746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:24.924017 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:24.924028 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.924786 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.924789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:24.924880 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:24.924902 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 13:45:24.924964 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 13:45:24.925031 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":3,"thread_start_us":86,"threads_started":1}
I20260926 13:45:24.925709 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:24.919373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60918 (local address 127.12.170.67:44257)
0926 13:45:24.919838 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:24.919850 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:24.920385 (+ 535us) server_negotiation.cc:407] Connection header received
0926 13:45:24.920593 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:24.920600 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:24.920804 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:24.920958 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:24.921248 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.922124 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.923746 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.924392 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.925312 (+ 920us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:24.925463 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:24.925501 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 13:45:24.925580 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":103,"threads_started":1}
I20260926 13:45:24.926465 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:24.920093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43108 (local address 127.12.170.66:42103)
0926 13:45:24.920807 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:24.920821 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:24.920838 (+ 17us) server_negotiation.cc:407] Connection header received
0926 13:45:24.920908 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:24.920912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:24.921046 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:24.921268 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:24.921429 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.922533 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.924175 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:24.924953 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:24.925737 (+ 784us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:24.925904 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:24.926095 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 13:45:24.926189 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":90,"threads_started":1}
I20260926 13:45:24.928398 20914 tablet_service.cc:1467] Processing CreateTablet for tablet 5f28eb4e77e2487b9ff8d096ff293da0 (DEFAULT_TABLE table=test-table [id=f2ba046223a74d84a3d9745aa0a42f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:24.928400 20634 tablet_service.cc:1467] Processing CreateTablet for tablet 5f28eb4e77e2487b9ff8d096ff293da0 (DEFAULT_TABLE table=test-table [id=f2ba046223a74d84a3d9745aa0a42f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:24.928602 20774 tablet_service.cc:1467] Processing CreateTablet for tablet 5f28eb4e77e2487b9ff8d096ff293da0 (DEFAULT_TABLE table=test-table [id=f2ba046223a74d84a3d9745aa0a42f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:24.929379 20774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f28eb4e77e2487b9ff8d096ff293da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:24.929404 20634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f28eb4e77e2487b9ff8d096ff293da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:24.929404 20914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f28eb4e77e2487b9ff8d096ff293da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:24.935804 21008 tablet_bootstrap.cc:492] T 5f28eb4e77e2487b9ff8d096ff293da0 P 556c77c7adf74f3b83543fea8130ed35: Bootstrap starting.
I20260926 13:45:24.937034 21010 tablet_bootstrap.cc:492] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3: Bootstrap starting.
I20260926 13:45:24.937893 21008 tablet_bootstrap.cc:654] T 5f28eb4e77e2487b9ff8d096ff293da0 P 556c77c7adf74f3b83543fea8130ed35: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:24.938582 21010 tablet_bootstrap.cc:654] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:24.938560 21008 log.cc:824] T 5f28eb4e77e2487b9ff8d096ff293da0 P 556c77c7adf74f3b83543fea8130ed35: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:24.939218 21010 log.cc:824] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:24.939832 21009 tablet_bootstrap.cc:492] T 5f28eb4e77e2487b9ff8d096ff293da0 P ff63674a2a8848d1a168dfbd580bbf3a: Bootstrap starting.
I20260926 13:45:24.940852 21008 tablet_bootstrap.cc:492] T 5f28eb4e77e2487b9ff8d096ff293da0 P 556c77c7adf74f3b83543fea8130ed35: No bootstrap required, opened a new log
I20260926 13:45:24.941046 21008 ts_tablet_manager.cc:1397] T 5f28eb4e77e2487b9ff8d096ff293da0 P 556c77c7adf74f3b83543fea8130ed35: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 13:45:24.941244 21010 tablet_bootstrap.cc:492] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3: No bootstrap required, opened a new log
I20260926 13:45:24.941465 21010 ts_tablet_manager.cc:1397] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 13:45:24.941666 21009 tablet_bootstrap.cc:654] T 5f28eb4e77e2487b9ff8d096ff293da0 P ff63674a2a8848d1a168dfbd580bbf3a: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:24.942373 21009 log.cc:824] T 5f28eb4e77e2487b9ff8d096ff293da0 P ff63674a2a8848d1a168dfbd580bbf3a: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:24.943950 21009 tablet_bootstrap.cc:492] T 5f28eb4e77e2487b9ff8d096ff293da0 P ff63674a2a8848d1a168dfbd580bbf3a: No bootstrap required, opened a new log
I20260926 13:45:24.944100 21009 ts_tablet_manager.cc:1397] T 5f28eb4e77e2487b9ff8d096ff293da0 P ff63674a2a8848d1a168dfbd580bbf3a: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260926 13:45:24.945152 21010 raft_consensus.cc:348] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } } peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } }
I20260926 13:45:24.945132 21008 raft_consensus.cc:348] T 5f28eb4e77e2487b9ff8d096ff293da0 P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } } peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } }
I20260926 13:45:24.945400 21008 raft_consensus.cc:374] T 5f28eb4e77e2487b9ff8d096ff293da0 P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:24.945400 21010 raft_consensus.cc:374] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:24.945459 21010 raft_consensus.cc:729] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0daec48270fb4665beb58e1cb1c672e3, State: Initialized, Role: FOLLOWER
I20260926 13:45:24.945459 21008 raft_consensus.cc:729] T 5f28eb4e77e2487b9ff8d096ff293da0 P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 556c77c7adf74f3b83543fea8130ed35, State: Initialized, Role: FOLLOWER
I20260926 13:45:24.945902 21008 consensus_queue.cc:260] T 5f28eb4e77e2487b9ff8d096ff293da0 P 556c77c7adf74f3b83543fea8130ed35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } } peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } }
I20260926 13:45:24.945899 21010 consensus_queue.cc:260] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } } peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } }
I20260926 13:45:24.946880 21008 ts_tablet_manager.cc:1428] T 5f28eb4e77e2487b9ff8d096ff293da0 P 556c77c7adf74f3b83543fea8130ed35: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 13:45:24.946880 21010 ts_tablet_manager.cc:1428] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 13:45:24.947227 20700 heartbeater.cc:503] Master 127.12.170.126:43289 was elected leader, sending a full tablet report...
I20260926 13:45:24.947090 21009 raft_consensus.cc:348] T 5f28eb4e77e2487b9ff8d096ff293da0 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } } peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } }
I20260926 13:45:24.947317 21009 raft_consensus.cc:374] T 5f28eb4e77e2487b9ff8d096ff293da0 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:24.947376 21009 raft_consensus.cc:729] T 5f28eb4e77e2487b9ff8d096ff293da0 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff63674a2a8848d1a168dfbd580bbf3a, State: Initialized, Role: FOLLOWER
I20260926 13:45:24.947467 20840 heartbeater.cc:503] Master 127.12.170.126:43289 was elected leader, sending a full tablet report...
I20260926 13:45:24.947750 21009 consensus_queue.cc:260] T 5f28eb4e77e2487b9ff8d096ff293da0 P ff63674a2a8848d1a168dfbd580bbf3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } } peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } }
I20260926 13:45:24.948279 20980 heartbeater.cc:503] Master 127.12.170.126:43289 was elected leader, sending a full tablet report...
I20260926 13:45:24.949671 21009 ts_tablet_manager.cc:1428] T 5f28eb4e77e2487b9ff8d096ff293da0 P ff63674a2a8848d1a168dfbd580bbf3a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260926 13:45:24.973495 20701 tablet.cc:2369] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:45:25.082633 21014 raft_consensus.cc:482] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:45:25.082857 21014 raft_consensus.cc:504] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } } peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } }
I20260926 13:45:25.083969 21014 leader_election.cc:288] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff63674a2a8848d1a168dfbd580bbf3a (127.12.170.67:44257), 556c77c7adf74f3b83543fea8130ed35 (127.12.170.66:42103)
I20260926 13:45:25.087510 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:25.084306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:52657 (local address 127.12.170.66:42103)
0926 13:45:25.084423 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:25.084428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:25.084563 (+ 135us) server_negotiation.cc:407] Connection header received
0926 13:45:25.084682 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:25.084684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:25.084721 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:25.084781 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:25.085028 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.085529 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.086625 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.087011 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.087113 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:25.087159 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:25.087317 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 13:45:25.087382 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 13:45:25.087517 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:25.084287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:51473 (local address 127.12.170.67:44257)
0926 13:45:25.084411 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:25.084416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:25.084538 (+ 122us) server_negotiation.cc:407] Connection header received
0926 13:45:25.084668 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:25.084671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:25.084710 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:25.084775 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:25.085052 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.085537 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.086615 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.087003 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.087098 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:25.087158 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:25.087330 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 13:45:25.087395 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 13:45:25.087548 21017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:25.084207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:42103 (local address 127.12.170.65:52657)
0926 13:45:25.084485 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:25.084496 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:25.084589 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:25.084800 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:25.084803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:25.084817 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:25.084934 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:25.084939 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.085629 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.085632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:25.086513 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:25.086521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.087115 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.087119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:25.087211 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:25.087231 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 13:45:25.087292 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 13:45:25.087408 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":53,"thread_start_us":57,"threads_started":1}
I20260926 13:45:25.087556 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:25.084159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:44257 (local address 127.12.170.65:51473)
0926 13:45:25.084459 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:25.084472 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:25.084571 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:25.084792 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:25.084794 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:25.084809 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:25.084953 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:25.084958 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.085632 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.085635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:25.086505 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:25.086513 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.087099 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.087102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:25.087191 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:25.087235 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 13:45:25.087303 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 13:45:25.087413 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":101,"spinlock_wait_cycles":18688,"thread_start_us":66,"threads_started":1}
I20260926 13:45:25.088164 20794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f28eb4e77e2487b9ff8d096ff293da0" candidate_uuid: "0daec48270fb4665beb58e1cb1c672e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "556c77c7adf74f3b83543fea8130ed35" is_pre_election: true
I20260926 13:45:25.088198 20934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f28eb4e77e2487b9ff8d096ff293da0" candidate_uuid: "0daec48270fb4665beb58e1cb1c672e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" is_pre_election: true
I20260926 13:45:25.088435 20794 raft_consensus.cc:2454] T 5f28eb4e77e2487b9ff8d096ff293da0 P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0daec48270fb4665beb58e1cb1c672e3 in term 0.
I20260926 13:45:25.088439 20934 raft_consensus.cc:2454] T 5f28eb4e77e2487b9ff8d096ff293da0 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0daec48270fb4665beb58e1cb1c672e3 in term 0.
I20260926 13:45:25.088848 20585 leader_election.cc:302] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0daec48270fb4665beb58e1cb1c672e3, ff63674a2a8848d1a168dfbd580bbf3a; no voters:
I20260926 13:45:25.089077 21014 raft_consensus.cc:2781] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:45:25.089155 21014 raft_consensus.cc:482] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:45:25.089201 21014 raft_consensus.cc:3037] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:25.090165 21014 raft_consensus.cc:504] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } } peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } }
I20260926 13:45:25.090451 21014 leader_election.cc:288] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3 [CANDIDATE]: Term 1 election: Requested vote from peers ff63674a2a8848d1a168dfbd580bbf3a (127.12.170.67:44257), 556c77c7adf74f3b83543fea8130ed35 (127.12.170.66:42103)
I20260926 13:45:25.090734 20934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f28eb4e77e2487b9ff8d096ff293da0" candidate_uuid: "0daec48270fb4665beb58e1cb1c672e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff63674a2a8848d1a168dfbd580bbf3a"
I20260926 13:45:25.090732 20794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f28eb4e77e2487b9ff8d096ff293da0" candidate_uuid: "0daec48270fb4665beb58e1cb1c672e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "556c77c7adf74f3b83543fea8130ed35"
I20260926 13:45:25.090854 20794 raft_consensus.cc:3037] T 5f28eb4e77e2487b9ff8d096ff293da0 P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:25.090849 20934 raft_consensus.cc:3037] T 5f28eb4e77e2487b9ff8d096ff293da0 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:25.092104 20794 raft_consensus.cc:2454] T 5f28eb4e77e2487b9ff8d096ff293da0 P 556c77c7adf74f3b83543fea8130ed35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0daec48270fb4665beb58e1cb1c672e3 in term 1.
I20260926 13:45:25.092353 20934 raft_consensus.cc:2454] T 5f28eb4e77e2487b9ff8d096ff293da0 P ff63674a2a8848d1a168dfbd580bbf3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0daec48270fb4665beb58e1cb1c672e3 in term 1.
I20260926 13:45:25.092391 20588 leader_election.cc:302] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0daec48270fb4665beb58e1cb1c672e3, 556c77c7adf74f3b83543fea8130ed35; no voters:
I20260926 13:45:25.092550 21014 raft_consensus.cc:2781] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:25.092733 21014 raft_consensus.cc:686] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3 [term 1 LEADER]: Becoming Leader. State: Replica: 0daec48270fb4665beb58e1cb1c672e3, State: Running, Role: LEADER
I20260926 13:45:25.093075 21014 consensus_queue.cc:237] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } } peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } }
I20260926 13:45:25.095894 20497 catalog_manager.cc:5495] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0daec48270fb4665beb58e1cb1c672e3 (127.12.170.65). New cstate: current_term: 1 leader_uuid: "0daec48270fb4665beb58e1cb1c672e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } health_report { overall_health: UNKNOWN } } }
I20260926 13:45:25.108902 21007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:25.105939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50648 (local address 127.12.170.65:45189)
0926 13:45:25.106042 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:25.106046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:25.106057 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:45:25.106154 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:25.106156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:25.106199 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:25.106276 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:45:25.106453 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.106938 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.107724 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.107954 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.108027 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:25.108214 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:45:25.108583 (+ 369us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:45:25.108658 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:25.108720 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 13:45:25.108784 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 13:45:25.114207 20934 raft_consensus.cc:1261] T 5f28eb4e77e2487b9ff8d096ff293da0 P ff63674a2a8848d1a168dfbd580bbf3a [term 1 FOLLOWER]: Refusing update from remote peer 0daec48270fb4665beb58e1cb1c672e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:25.114504 20794 raft_consensus.cc:1261] T 5f28eb4e77e2487b9ff8d096ff293da0 P 556c77c7adf74f3b83543fea8130ed35 [term 1 FOLLOWER]: Refusing update from remote peer 0daec48270fb4665beb58e1cb1c672e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260926 13:45:25.114799 20981 tablet.cc:2369] T 5f28eb4e77e2487b9ff8d096ff293da0 P ff63674a2a8848d1a168dfbd580bbf3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:45:25.114862 21019 consensus_queue.cc:1035] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:25.115048 21014 consensus_queue.cc:1035] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:25.121984 21023 mvcc.cc:205] Tried to move back new op lower bound from 7333602611660873728 to 7333602611583475712. Current Snapshot: MvccSnapshot[applied={T|T < 7333602611660873728}]
I20260926 13:45:25.123574 21024 mvcc.cc:205] Tried to move back new op lower bound from 7333602611660873728 to 7333602611583475712. Current Snapshot: MvccSnapshot[applied={T|T < 7333602611660873728}]
I20260926 13:45:25.129691 20497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:45:25.129813 20497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:45:25.131955 20497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e: Sending DeleteTablet for 3 replicas of tablet 5f28eb4e77e2487b9ff8d096ff293da0
I20260926 13:45:25.132613 20914 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f28eb4e77e2487b9ff8d096ff293da0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:25 UTC) from {username='oryx', principal='oryx/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:60918
I20260926 13:45:25.132709 20634 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f28eb4e77e2487b9ff8d096ff293da0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:25 UTC) from {username='oryx', principal='oryx/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:50642
I20260926 13:45:25.132964 20774 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f28eb4e77e2487b9ff8d096ff293da0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:25 UTC) from {username='oryx', principal='oryx/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:43108
I20260926 13:45:25.133078 21032 tablet_replica.cc:330] T 5f28eb4e77e2487b9ff8d096ff293da0 P ff63674a2a8848d1a168dfbd580bbf3a: stopping tablet replica
I20260926 13:45:25.133329 21034 tablet_replica.cc:330] T 5f28eb4e77e2487b9ff8d096ff293da0 P 556c77c7adf74f3b83543fea8130ed35: stopping tablet replica
I20260926 13:45:25.133361 21032 raft_consensus.cc:2229] T 5f28eb4e77e2487b9ff8d096ff293da0 P ff63674a2a8848d1a168dfbd580bbf3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:45:25.133630 21032 raft_consensus.cc:2258] T 5f28eb4e77e2487b9ff8d096ff293da0 P ff63674a2a8848d1a168dfbd580bbf3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:25.133658 21034 raft_consensus.cc:2229] T 5f28eb4e77e2487b9ff8d096ff293da0 P 556c77c7adf74f3b83543fea8130ed35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:45:25.133824 20497 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:45:25.134199 20497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:45:25.134402 21034 raft_consensus.cc:2258] T 5f28eb4e77e2487b9ff8d096ff293da0 P 556c77c7adf74f3b83543fea8130ed35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:25.135033 21032 ts_tablet_manager.cc:1905] T 5f28eb4e77e2487b9ff8d096ff293da0 P ff63674a2a8848d1a168dfbd580bbf3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:25.136041 21034 ts_tablet_manager.cc:1905] T 5f28eb4e77e2487b9ff8d096ff293da0 P 556c77c7adf74f3b83543fea8130ed35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:25.137759 21032 ts_tablet_manager.cc:1918] T 5f28eb4e77e2487b9ff8d096ff293da0 P ff63674a2a8848d1a168dfbd580bbf3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:45:25.137861 21032 log.cc:1196] T 5f28eb4e77e2487b9ff8d096ff293da0 P ff63674a2a8848d1a168dfbd580bbf3a: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/5f28eb4e77e2487b9ff8d096ff293da0
I20260926 13:45:25.138185 21032 ts_tablet_manager.cc:1939] T 5f28eb4e77e2487b9ff8d096ff293da0 P ff63674a2a8848d1a168dfbd580bbf3a: Deleting consensus metadata
I20260926 13:45:25.138765 21034 ts_tablet_manager.cc:1918] T 5f28eb4e77e2487b9ff8d096ff293da0 P 556c77c7adf74f3b83543fea8130ed35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:45:25.138908 21034 log.cc:1196] T 5f28eb4e77e2487b9ff8d096ff293da0 P 556c77c7adf74f3b83543fea8130ed35: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-1/wal/wals/5f28eb4e77e2487b9ff8d096ff293da0
I20260926 13:45:25.139266 21034 ts_tablet_manager.cc:1939] T 5f28eb4e77e2487b9ff8d096ff293da0 P 556c77c7adf74f3b83543fea8130ed35: Deleting consensus metadata
I20260926 13:45:25.139640 20481 catalog_manager.cc:4841] TS ff63674a2a8848d1a168dfbd580bbf3a (127.12.170.67:44257): tablet 5f28eb4e77e2487b9ff8d096ff293da0 (table test-table [id=f2ba046223a74d84a3d9745aa0a42f49]) successfully deleted
I20260926 13:45:25.140054 20914 tablet_service.cc:1467] Processing CreateTablet for tablet 0bfadc3f8869420598728ba25ab2d5b5 (DEFAULT_TABLE table=test-table [id=105cae234bfb4378ae3a0dd3b94e8559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:25.140285 20914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bfadc3f8869420598728ba25ab2d5b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:25.141919 21009 tablet_bootstrap.cc:492] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a: Bootstrap starting.
I20260926 13:45:25.142081 20634 tablet_service.cc:1467] Processing CreateTablet for tablet 0bfadc3f8869420598728ba25ab2d5b5 (DEFAULT_TABLE table=test-table [id=105cae234bfb4378ae3a0dd3b94e8559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:25.142364 20484 catalog_manager.cc:4841] TS 556c77c7adf74f3b83543fea8130ed35 (127.12.170.66:42103): tablet 5f28eb4e77e2487b9ff8d096ff293da0 (table test-table [id=f2ba046223a74d84a3d9745aa0a42f49]) successfully deleted
I20260926 13:45:25.142655 20774 tablet_service.cc:1467] Processing CreateTablet for tablet 0bfadc3f8869420598728ba25ab2d5b5 (DEFAULT_TABLE table=test-table [id=105cae234bfb4378ae3a0dd3b94e8559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:25.142854 20774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bfadc3f8869420598728ba25ab2d5b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:25.142966 21009 tablet_bootstrap.cc:654] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:25.144152 21009 tablet_bootstrap.cc:492] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a: No bootstrap required, opened a new log
I20260926 13:45:25.144233 21009 ts_tablet_manager.cc:1397] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:25.144644 21009 raft_consensus.cc:348] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.144709 21008 tablet_bootstrap.cc:492] T 0bfadc3f8869420598728ba25ab2d5b5 P 556c77c7adf74f3b83543fea8130ed35: Bootstrap starting.
I20260926 13:45:25.144747 21009 raft_consensus.cc:374] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:25.144781 21009 raft_consensus.cc:729] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff63674a2a8848d1a168dfbd580bbf3a, State: Initialized, Role: FOLLOWER
I20260926 13:45:25.144893 21009 consensus_queue.cc:260] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.145133 21009 ts_tablet_manager.cc:1428] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:45:25.145509 20634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bfadc3f8869420598728ba25ab2d5b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:25.145633 21008 tablet_bootstrap.cc:654] T 0bfadc3f8869420598728ba25ab2d5b5 P 556c77c7adf74f3b83543fea8130ed35: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:25.146785 21008 tablet_bootstrap.cc:492] T 0bfadc3f8869420598728ba25ab2d5b5 P 556c77c7adf74f3b83543fea8130ed35: No bootstrap required, opened a new log
I20260926 13:45:25.146873 21008 ts_tablet_manager.cc:1397] T 0bfadc3f8869420598728ba25ab2d5b5 P 556c77c7adf74f3b83543fea8130ed35: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:25.147310 21008 raft_consensus.cc:348] T 0bfadc3f8869420598728ba25ab2d5b5 P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.147415 21008 raft_consensus.cc:374] T 0bfadc3f8869420598728ba25ab2d5b5 P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:25.147447 21008 raft_consensus.cc:729] T 0bfadc3f8869420598728ba25ab2d5b5 P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 556c77c7adf74f3b83543fea8130ed35, State: Initialized, Role: FOLLOWER
I20260926 13:45:25.147495 21010 tablet_bootstrap.cc:492] T 0bfadc3f8869420598728ba25ab2d5b5 P 0daec48270fb4665beb58e1cb1c672e3: Bootstrap starting.
I20260926 13:45:25.147555 21008 consensus_queue.cc:260] T 0bfadc3f8869420598728ba25ab2d5b5 P 556c77c7adf74f3b83543fea8130ed35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.147893 21008 ts_tablet_manager.cc:1428] T 0bfadc3f8869420598728ba25ab2d5b5 P 556c77c7adf74f3b83543fea8130ed35: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 13:45:25.148376 21010 tablet_bootstrap.cc:654] T 0bfadc3f8869420598728ba25ab2d5b5 P 0daec48270fb4665beb58e1cb1c672e3: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:25.149925 21010 tablet_bootstrap.cc:492] T 0bfadc3f8869420598728ba25ab2d5b5 P 0daec48270fb4665beb58e1cb1c672e3: No bootstrap required, opened a new log
I20260926 13:45:25.150007 21010 ts_tablet_manager.cc:1397] T 0bfadc3f8869420598728ba25ab2d5b5 P 0daec48270fb4665beb58e1cb1c672e3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 13:45:25.150444 21010 raft_consensus.cc:348] T 0bfadc3f8869420598728ba25ab2d5b5 P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.150552 21010 raft_consensus.cc:374] T 0bfadc3f8869420598728ba25ab2d5b5 P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:25.150568 21033 tablet_replica.cc:330] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3: stopping tablet replica
I20260926 13:45:25.150594 21010 raft_consensus.cc:729] T 0bfadc3f8869420598728ba25ab2d5b5 P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0daec48270fb4665beb58e1cb1c672e3, State: Initialized, Role: FOLLOWER
I20260926 13:45:25.150712 21010 consensus_queue.cc:260] T 0bfadc3f8869420598728ba25ab2d5b5 P 0daec48270fb4665beb58e1cb1c672e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.150823 21033 raft_consensus.cc:2229] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:45:25.150978 21010 ts_tablet_manager.cc:1428] T 0bfadc3f8869420598728ba25ab2d5b5 P 0daec48270fb4665beb58e1cb1c672e3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 13:45:25.151198 21033 raft_consensus.cc:2258] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:25.152668 21033 ts_tablet_manager.cc:1905] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:25.154742 21033 ts_tablet_manager.cc:1918] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:45:25.154810 21033 log.cc:1196] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-0/wal/wals/5f28eb4e77e2487b9ff8d096ff293da0
I20260926 13:45:25.155041 21033 ts_tablet_manager.cc:1939] T 5f28eb4e77e2487b9ff8d096ff293da0 P 0daec48270fb4665beb58e1cb1c672e3: Deleting consensus metadata
I20260926 13:45:25.156064 20481 catalog_manager.cc:4841] TS 0daec48270fb4665beb58e1cb1c672e3 (127.12.170.65:45189): tablet 5f28eb4e77e2487b9ff8d096ff293da0 (table test-table [id=f2ba046223a74d84a3d9745aa0a42f49]) successfully deleted
W20260926 13:45:25.168226 20841 tablet.cc:2369] T 0bfadc3f8869420598728ba25ab2d5b5 P 556c77c7adf74f3b83543fea8130ed35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:45:25.344099 21016 raft_consensus.cc:482] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:45:25.344231 21016 raft_consensus.cc:504] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.345221 21016 leader_election.cc:288] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0daec48270fb4665beb58e1cb1c672e3 (127.12.170.65:45189), 556c77c7adf74f3b83543fea8130ed35 (127.12.170.66:42103)
I20260926 13:45:25.348753 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:25.345440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:42103 (local address 127.12.170.67:54349)
0926 13:45:25.345540 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:25.345555 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:25.345640 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:25.345876 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:25.345878 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:25.345896 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:25.346016 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:25.346023 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.346809 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.346813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:25.347710 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:25.347721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.348366 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.348370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:25.348493 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:25.348519 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 13:45:25.348584 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 13:45:25.348649 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":2}
I20260926 13:45:25.348790 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:25.345555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:54349 (local address 127.12.170.66:42103)
0926 13:45:25.345662 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:25.345666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:25.345680 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:45:25.345735 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:25.345738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:25.345783 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:25.345869 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:25.346121 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.346713 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.347820 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.348271 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.348356 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:25.348394 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:25.348606 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 13:45:25.348673 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 13:45:25.348976 21007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:25.345537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:39139 (local address 127.12.170.65:45189)
0926 13:45:25.345660 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:25.345664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:25.345933 (+ 269us) server_negotiation.cc:407] Connection header received
0926 13:45:25.346043 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:25.346046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:25.346097 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:25.346161 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:25.346423 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.346956 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.348040 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.348461 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.348567 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:25.348611 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:25.348794 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 13:45:25.348861 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 13:45:25.349110 21038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:25.345444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:45189 (local address 127.12.170.67:39139)
0926 13:45:25.345839 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:25.345853 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:25.345956 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:25.346182 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:25.346184 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:25.346200 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:25.346322 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:25.346328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.347055 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.347058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:25.347934 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:25.347943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.348581 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.348586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:25.348677 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:25.348698 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 13:45:25.348769 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 13:45:25.348974 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":140,"thread_start_us":65,"threads_started":1}
I20260926 13:45:25.349352 20794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bfadc3f8869420598728ba25ab2d5b5" candidate_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "556c77c7adf74f3b83543fea8130ed35" is_pre_election: true
I20260926 13:45:25.349475 20794 raft_consensus.cc:2454] T 0bfadc3f8869420598728ba25ab2d5b5 P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff63674a2a8848d1a168dfbd580bbf3a in term 0.
I20260926 13:45:25.349572 20654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bfadc3f8869420598728ba25ab2d5b5" candidate_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0daec48270fb4665beb58e1cb1c672e3" is_pre_election: true
I20260926 13:45:25.349800 20654 raft_consensus.cc:2454] T 0bfadc3f8869420598728ba25ab2d5b5 P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff63674a2a8848d1a168dfbd580bbf3a in term 0.
I20260926 13:45:25.349798 20868 leader_election.cc:302] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 556c77c7adf74f3b83543fea8130ed35, ff63674a2a8848d1a168dfbd580bbf3a; no voters:
I20260926 13:45:25.350021 21016 raft_consensus.cc:2781] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:45:25.350080 21016 raft_consensus.cc:482] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:45:25.350111 21016 raft_consensus.cc:3037] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:25.351029 21016 raft_consensus.cc:504] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.351325 21016 leader_election.cc:288] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a [CANDIDATE]: Term 1 election: Requested vote from peers 0daec48270fb4665beb58e1cb1c672e3 (127.12.170.65:45189), 556c77c7adf74f3b83543fea8130ed35 (127.12.170.66:42103)
I20260926 13:45:25.351703 20654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bfadc3f8869420598728ba25ab2d5b5" candidate_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0daec48270fb4665beb58e1cb1c672e3"
I20260926 13:45:25.351692 20794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bfadc3f8869420598728ba25ab2d5b5" candidate_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "556c77c7adf74f3b83543fea8130ed35"
I20260926 13:45:25.351823 20654 raft_consensus.cc:3037] T 0bfadc3f8869420598728ba25ab2d5b5 P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:25.351823 20794 raft_consensus.cc:3037] T 0bfadc3f8869420598728ba25ab2d5b5 P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:25.352924 20654 raft_consensus.cc:2454] T 0bfadc3f8869420598728ba25ab2d5b5 P 0daec48270fb4665beb58e1cb1c672e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff63674a2a8848d1a168dfbd580bbf3a in term 1.
I20260926 13:45:25.352924 20794 raft_consensus.cc:2454] T 0bfadc3f8869420598728ba25ab2d5b5 P 556c77c7adf74f3b83543fea8130ed35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff63674a2a8848d1a168dfbd580bbf3a in term 1.
I20260926 13:45:25.353186 20868 leader_election.cc:302] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 556c77c7adf74f3b83543fea8130ed35, ff63674a2a8848d1a168dfbd580bbf3a; no voters:
I20260926 13:45:25.353333 21016 raft_consensus.cc:2781] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:25.353508 21016 raft_consensus.cc:686] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a [term 1 LEADER]: Becoming Leader. State: Replica: ff63674a2a8848d1a168dfbd580bbf3a, State: Running, Role: LEADER
I20260926 13:45:25.353716 21016 consensus_queue.cc:237] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.355772 20497 catalog_manager.cc:5495] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a reported cstate change: term changed from 0 to 1, leader changed from <none> to ff63674a2a8848d1a168dfbd580bbf3a (127.12.170.67). New cstate: current_term: 1 leader_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } health_report { overall_health: HEALTHY } } }
Sep 26 13:45:25 dist-test-slave-hc1k krb5kdc[20452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430324, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.126@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM
I20260926 13:45:25.412667 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:25.405209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:43289 (local address 127.0.0.1:36618)
0926 13:45:25.405484 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:25.405496 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:25.405601 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:25.405841 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:25.405844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:25.406242 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:25.406339 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:25.406345 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.407079 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.407083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:25.407629 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:25.407636 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.408038 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.408041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:25.408124 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:25.408598 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:25.408609 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:25.409897 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:25.411418 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:25.411426 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:25.411430 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:25.411641 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:25.411873 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:25.411875 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:25.411876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:25.411947 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:25.412224 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:25.412229 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:25.412324 (+ 95us) client_negotiation.cc:769] Sending connection context
0926 13:45:25.412409 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 13:45:25.412518 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":1,"thread_start_us":110,"threads_started":1}
I20260926 13:45:25.412688 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:25.405269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.12.170.126:43289)
0926 13:45:25.405386 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:25.405391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:25.405564 (+ 173us) server_negotiation.cc:407] Connection header received
0926 13:45:25.405681 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:25.405684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:25.405719 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:25.405824 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:25.406443 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.406984 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.407748 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.407952 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.408032 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:25.410023 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:25.410026 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:25.410028 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:25.410054 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:25.411326 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:25.411737 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:25.411739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:25.411741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:25.411810 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:25.412005 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:25.412007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:25.412008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:25.412157 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:25.412223 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:25.412444 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 13:45:25.412569 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":42}
I20260926 13:45:25.415948 20497 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:36618:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 13:45:25.416607 20497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:45:25.420925 20774 tablet_service.cc:1467] Processing CreateTablet for tablet 6b65ae213a034dc49cd2f450bf38a96a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=831d4740fa2d443098af5fec688c748f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:45:25.420936 20634 tablet_service.cc:1467] Processing CreateTablet for tablet 6b65ae213a034dc49cd2f450bf38a96a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=831d4740fa2d443098af5fec688c748f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:45:25.421046 20914 tablet_service.cc:1467] Processing CreateTablet for tablet 6b65ae213a034dc49cd2f450bf38a96a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=831d4740fa2d443098af5fec688c748f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:45:25.421267 20774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b65ae213a034dc49cd2f450bf38a96a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:25.421293 20634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b65ae213a034dc49cd2f450bf38a96a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:25.421350 20914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b65ae213a034dc49cd2f450bf38a96a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:25.423527 21008 tablet_bootstrap.cc:492] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35: Bootstrap starting.
I20260926 13:45:25.423871 21010 tablet_bootstrap.cc:492] T 6b65ae213a034dc49cd2f450bf38a96a P 0daec48270fb4665beb58e1cb1c672e3: Bootstrap starting.
I20260926 13:45:25.424304 21009 tablet_bootstrap.cc:492] T 6b65ae213a034dc49cd2f450bf38a96a P ff63674a2a8848d1a168dfbd580bbf3a: Bootstrap starting.
I20260926 13:45:25.424487 21008 tablet_bootstrap.cc:654] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:25.424608 21010 tablet_bootstrap.cc:654] T 6b65ae213a034dc49cd2f450bf38a96a P 0daec48270fb4665beb58e1cb1c672e3: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:25.424949 21009 tablet_bootstrap.cc:654] T 6b65ae213a034dc49cd2f450bf38a96a P ff63674a2a8848d1a168dfbd580bbf3a: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:25.425671 21008 tablet_bootstrap.cc:492] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35: No bootstrap required, opened a new log
I20260926 13:45:25.425748 21008 ts_tablet_manager.cc:1397] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 13:45:25.426169 21009 tablet_bootstrap.cc:492] T 6b65ae213a034dc49cd2f450bf38a96a P ff63674a2a8848d1a168dfbd580bbf3a: No bootstrap required, opened a new log
I20260926 13:45:25.426152 21008 raft_consensus.cc:348] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.426242 21009 ts_tablet_manager.cc:1397] T 6b65ae213a034dc49cd2f450bf38a96a P ff63674a2a8848d1a168dfbd580bbf3a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:25.426267 21008 raft_consensus.cc:374] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:25.426303 21008 raft_consensus.cc:729] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 556c77c7adf74f3b83543fea8130ed35, State: Initialized, Role: FOLLOWER
I20260926 13:45:25.426420 21008 consensus_queue.cc:260] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.426623 21009 raft_consensus.cc:348] T 6b65ae213a034dc49cd2f450bf38a96a P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.426643 21015 tablet_replica.cc:439] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } } }
I20260926 13:45:25.426721 21009 raft_consensus.cc:374] T 6b65ae213a034dc49cd2f450bf38a96a P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:25.426753 21009 raft_consensus.cc:729] T 6b65ae213a034dc49cd2f450bf38a96a P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff63674a2a8848d1a168dfbd580bbf3a, State: Initialized, Role: FOLLOWER
I20260926 13:45:25.426658 21008 ts_tablet_manager.cc:1428] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 13:45:25.426774 21015 tablet_replica.cc:442] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:45:25.426854 21009 consensus_queue.cc:260] T 6b65ae213a034dc49cd2f450bf38a96a P ff63674a2a8848d1a168dfbd580bbf3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.427109 21009 ts_tablet_manager.cc:1428] T 6b65ae213a034dc49cd2f450bf38a96a P ff63674a2a8848d1a168dfbd580bbf3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:45:25.427081 21016 tablet_replica.cc:439] T 6b65ae213a034dc49cd2f450bf38a96a P ff63674a2a8848d1a168dfbd580bbf3a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } } }
I20260926 13:45:25.427196 21016 tablet_replica.cc:442] T 6b65ae213a034dc49cd2f450bf38a96a P ff63674a2a8848d1a168dfbd580bbf3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:45:25.427477 21010 tablet_bootstrap.cc:492] T 6b65ae213a034dc49cd2f450bf38a96a P 0daec48270fb4665beb58e1cb1c672e3: No bootstrap required, opened a new log
I20260926 13:45:25.427554 21010 ts_tablet_manager.cc:1397] T 6b65ae213a034dc49cd2f450bf38a96a P 0daec48270fb4665beb58e1cb1c672e3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 13:45:25.427959 21010 raft_consensus.cc:348] T 6b65ae213a034dc49cd2f450bf38a96a P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.428062 21010 raft_consensus.cc:374] T 6b65ae213a034dc49cd2f450bf38a96a P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:25.428094 21010 raft_consensus.cc:729] T 6b65ae213a034dc49cd2f450bf38a96a P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0daec48270fb4665beb58e1cb1c672e3, State: Initialized, Role: FOLLOWER
I20260926 13:45:25.428203 21010 consensus_queue.cc:260] T 6b65ae213a034dc49cd2f450bf38a96a P 0daec48270fb4665beb58e1cb1c672e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.428429 21010 ts_tablet_manager.cc:1428] T 6b65ae213a034dc49cd2f450bf38a96a P 0daec48270fb4665beb58e1cb1c672e3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:45:25.428597 21027 tablet_replica.cc:439] T 6b65ae213a034dc49cd2f450bf38a96a P 0daec48270fb4665beb58e1cb1c672e3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } } }
I20260926 13:45:25.428709 21027 tablet_replica.cc:442] T 6b65ae213a034dc49cd2f450bf38a96a P 0daec48270fb4665beb58e1cb1c672e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:45:25.648257 21015 raft_consensus.cc:482] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:45:25.648378 21015 raft_consensus.cc:504] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.649430 21015 leader_election.cc:288] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0daec48270fb4665beb58e1cb1c672e3 (127.12.170.65:45189), ff63674a2a8848d1a168dfbd580bbf3a (127.12.170.67:44257)
I20260926 13:45:25.653769 21007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:25.649714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:46825 (local address 127.12.170.65:45189)
0926 13:45:25.649854 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:25.649859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:25.649963 (+ 104us) server_negotiation.cc:407] Connection header received
0926 13:45:25.650110 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:25.650114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:25.650163 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:25.650242 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:25.650517 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.651108 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.652552 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.653146 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.653262 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:25.653327 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:25.653579 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 13:45:25.653649 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 13:45:25.653852 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:25.650000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:44257 (local address 127.12.170.66:40465)
0926 13:45:25.650248 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:25.650260 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:25.650353 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:25.650572 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:25.650575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:25.650591 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:25.650694 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:25.650700 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.651469 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.651472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:25.652572 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:25.652579 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.653380 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.653384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:25.653491 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:25.653512 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 13:45:25.653579 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 13:45:25.653647 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":6,"thread_start_us":55,"threads_started":1}
I20260926 13:45:25.653865 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:25.649651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:45189 (local address 127.12.170.66:46825)
0926 13:45:25.649870 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:25.649883 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:25.650011 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:25.650260 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:25.650264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:25.650289 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:25.650416 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:25.650422 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.651207 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.651210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:25.652445 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:25.652455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.653284 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.653287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:25.653442 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:25.653472 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 13:45:25.653568 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 13:45:25.653647 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":26,"thread_start_us":59,"threads_started":1}
I20260926 13:45:25.654232 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:25.650094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:40465 (local address 127.12.170.67:44257)
0926 13:45:25.650267 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:25.650270 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:25.650331 (+ 61us) server_negotiation.cc:407] Connection header received
0926 13:45:25.650435 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:25.650437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:25.650484 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:25.650558 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:25.650785 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.651374 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.652667 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.653264 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.653368 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:25.653406 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:25.654037 (+ 631us) server_negotiation.cc:299] Negotiation successful
0926 13:45:25.654111 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":52,"threads_started":1}
I20260926 13:45:25.654615 20654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b65ae213a034dc49cd2f450bf38a96a" candidate_uuid: "556c77c7adf74f3b83543fea8130ed35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0daec48270fb4665beb58e1cb1c672e3" is_pre_election: true
I20260926 13:45:25.654640 20934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b65ae213a034dc49cd2f450bf38a96a" candidate_uuid: "556c77c7adf74f3b83543fea8130ed35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" is_pre_election: true
I20260926 13:45:25.654747 20934 raft_consensus.cc:2454] T 6b65ae213a034dc49cd2f450bf38a96a P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 556c77c7adf74f3b83543fea8130ed35 in term 0.
I20260926 13:45:25.654795 20654 raft_consensus.cc:2454] T 6b65ae213a034dc49cd2f450bf38a96a P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 556c77c7adf74f3b83543fea8130ed35 in term 0.
I20260926 13:45:25.655069 20725 leader_election.cc:302] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 556c77c7adf74f3b83543fea8130ed35, ff63674a2a8848d1a168dfbd580bbf3a; no voters:
I20260926 13:45:25.655293 21015 raft_consensus.cc:2781] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:45:25.655350 21015 raft_consensus.cc:482] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:45:25.655391 21015 raft_consensus.cc:3037] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:25.656360 21015 raft_consensus.cc:504] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.656633 21015 leader_election.cc:288] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35 [CANDIDATE]: Term 1 election: Requested vote from peers 0daec48270fb4665beb58e1cb1c672e3 (127.12.170.65:45189), ff63674a2a8848d1a168dfbd580bbf3a (127.12.170.67:44257)
I20260926 13:45:25.656988 20934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b65ae213a034dc49cd2f450bf38a96a" candidate_uuid: "556c77c7adf74f3b83543fea8130ed35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff63674a2a8848d1a168dfbd580bbf3a"
I20260926 13:45:25.656991 20654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b65ae213a034dc49cd2f450bf38a96a" candidate_uuid: "556c77c7adf74f3b83543fea8130ed35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0daec48270fb4665beb58e1cb1c672e3"
I20260926 13:45:25.657083 20934 raft_consensus.cc:3037] T 6b65ae213a034dc49cd2f450bf38a96a P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:25.657109 20654 raft_consensus.cc:3037] T 6b65ae213a034dc49cd2f450bf38a96a P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:25.657856 20934 raft_consensus.cc:2454] T 6b65ae213a034dc49cd2f450bf38a96a P ff63674a2a8848d1a168dfbd580bbf3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 556c77c7adf74f3b83543fea8130ed35 in term 1.
I20260926 13:45:25.658004 20654 raft_consensus.cc:2454] T 6b65ae213a034dc49cd2f450bf38a96a P 0daec48270fb4665beb58e1cb1c672e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 556c77c7adf74f3b83543fea8130ed35 in term 1.
I20260926 13:45:25.658113 20725 leader_election.cc:302] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 556c77c7adf74f3b83543fea8130ed35, ff63674a2a8848d1a168dfbd580bbf3a; no voters:
I20260926 13:45:25.658282 21015 raft_consensus.cc:2781] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:25.658430 21015 raft_consensus.cc:686] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35 [term 1 LEADER]: Becoming Leader. State: Replica: 556c77c7adf74f3b83543fea8130ed35, State: Running, Role: LEADER
I20260926 13:45:25.658660 21015 consensus_queue.cc:237] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.659750 21053 tablet_replica.cc:439] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35: TxnStatusTablet state changed. Reason: New leader 556c77c7adf74f3b83543fea8130ed35. Latest consensus state: current_term: 1 leader_uuid: "556c77c7adf74f3b83543fea8130ed35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } } }
I20260926 13:45:25.659869 21053 tablet_replica.cc:442] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:45:25.660107 21055 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 13:45:25.660246 21055 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 13:45:25.660880 20495 catalog_manager.cc:5495] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 556c77c7adf74f3b83543fea8130ed35 (127.12.170.66). New cstate: current_term: 1 leader_uuid: "556c77c7adf74f3b83543fea8130ed35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } health_report { overall_health: UNKNOWN } } }
I20260926 13:45:25.719961 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:25.716492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43116 (local address 127.12.170.66:42103)
0926 13:45:25.716608 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:25.716612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:25.716710 (+ 98us) server_negotiation.cc:407] Connection header received
0926 13:45:25.716896 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:25.716899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:25.716952 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:25.717030 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:45:25.717265 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.717770 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.718586 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.718807 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.718894 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:25.719178 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:45:25.719560 (+ 382us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:45:25.719690 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:25.719772 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 13:45:25.719823 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 13:45:25.719961 21046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:25.716396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:42103 (local address 127.0.0.1:43116)
0926 13:45:25.716639 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:25.716654 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:25.716764 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:25.717057 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:25.717060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:25.717067 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:45:25.717167 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:25.717173 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.717871 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.717874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:25.718476 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:25.718484 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.718913 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.718917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:25.719008 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:25.719049 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:45:25.719670 (+ 621us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:45:25.719685 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 13:45:25.719743 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 13:45:25.719823 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260926 13:45:25.722963 20934 raft_consensus.cc:1261] T 6b65ae213a034dc49cd2f450bf38a96a P ff63674a2a8848d1a168dfbd580bbf3a [term 1 FOLLOWER]: Refusing update from remote peer 556c77c7adf74f3b83543fea8130ed35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:25.723088 20654 raft_consensus.cc:1261] T 6b65ae213a034dc49cd2f450bf38a96a P 0daec48270fb4665beb58e1cb1c672e3 [term 1 FOLLOWER]: Refusing update from remote peer 556c77c7adf74f3b83543fea8130ed35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:25.723443 21053 consensus_queue.cc:1035] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:25.723598 21015 consensus_queue.cc:1035] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:25.725128 21016 tablet_replica.cc:439] T 6b65ae213a034dc49cd2f450bf38a96a P ff63674a2a8848d1a168dfbd580bbf3a: TxnStatusTablet state changed. Reason: New leader 556c77c7adf74f3b83543fea8130ed35. Latest consensus state: current_term: 1 leader_uuid: "556c77c7adf74f3b83543fea8130ed35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } } }
I20260926 13:45:25.725234 21016 tablet_replica.cc:442] T 6b65ae213a034dc49cd2f450bf38a96a P ff63674a2a8848d1a168dfbd580bbf3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:45:25.725258 21027 tablet_replica.cc:439] T 6b65ae213a034dc49cd2f450bf38a96a P 0daec48270fb4665beb58e1cb1c672e3: TxnStatusTablet state changed. Reason: New leader 556c77c7adf74f3b83543fea8130ed35. Latest consensus state: current_term: 1 leader_uuid: "556c77c7adf74f3b83543fea8130ed35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } } }
I20260926 13:45:25.725368 21027 tablet_replica.cc:442] T 6b65ae213a034dc49cd2f450bf38a96a P 0daec48270fb4665beb58e1cb1c672e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:45:25.727150 21015 tablet_replica.cc:439] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "556c77c7adf74f3b83543fea8130ed35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } } }
I20260926 13:45:25.727262 21015 tablet_replica.cc:442] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:45:25.728946 21027 tablet_replica.cc:439] T 6b65ae213a034dc49cd2f450bf38a96a P 0daec48270fb4665beb58e1cb1c672e3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "556c77c7adf74f3b83543fea8130ed35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } } }
I20260926 13:45:25.729046 21027 tablet_replica.cc:442] T 6b65ae213a034dc49cd2f450bf38a96a P 0daec48270fb4665beb58e1cb1c672e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:45:25.729579 21053 tablet_replica.cc:439] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "556c77c7adf74f3b83543fea8130ed35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } } }
I20260926 13:45:25.729673 21053 tablet_replica.cc:442] T 6b65ae213a034dc49cd2f450bf38a96a P 556c77c7adf74f3b83543fea8130ed35: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:45:25.730669 21016 tablet_replica.cc:439] T 6b65ae213a034dc49cd2f450bf38a96a P ff63674a2a8848d1a168dfbd580bbf3a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "556c77c7adf74f3b83543fea8130ed35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } } }
I20260926 13:45:25.730779 21016 tablet_replica.cc:442] T 6b65ae213a034dc49cd2f450bf38a96a P ff63674a2a8848d1a168dfbd580bbf3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:45:25.736974 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:25.733500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60922 (local address 127.12.170.67:44257)
0926 13:45:25.733600 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:25.733604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:25.733736 (+ 132us) server_negotiation.cc:407] Connection header received
0926 13:45:25.733882 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:25.733884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:25.733926 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:25.733995 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:45:25.734199 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.734671 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.735833 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.736033 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.736101 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:25.736330 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:45:25.736670 (+ 340us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:45:25.736745 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:25.736832 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 13:45:25.736874 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 13:45:25.746387 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:25.743160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:45831 (local address 127.12.170.66:42103)
0926 13:45:25.743255 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:25.743268 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:25.743501 (+ 233us) server_negotiation.cc:407] Connection header received
0926 13:45:25.743600 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:25.743602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:25.743637 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:25.743715 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:45:25.743986 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.744431 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.745543 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.745757 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.745829 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:25.746082 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:45:25.746135 (+ 53us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:45:25.746178 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:25.746268 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 13:45:25.746303 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 13:45:25.746455 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:25.743073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:42103 (local address 127.12.170.67:45831)
0926 13:45:25.743421 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:25.743434 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:25.743529 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:25.743722 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:25.743725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:25.743733 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:45:25.743900 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:25.743907 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.744531 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.744535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:25.745423 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:25.745433 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.745857 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.745860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:25.745938 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:25.745982 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:45:25.746197 (+ 215us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:45:25.746202 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:45:25.746257 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 13:45:25.746327 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":91,"threads_started":1}
I20260926 13:45:25.751878 21016 consensus_queue.cc:1035] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:25.752107 20794 raft_consensus.cc:1261] T 0bfadc3f8869420598728ba25ab2d5b5 P 556c77c7adf74f3b83543fea8130ed35 [term 1 FOLLOWER]: Refusing update from remote peer ff63674a2a8848d1a168dfbd580bbf3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:25.752475 21016 consensus_queue.cc:1035] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:25.755914 21061 mvcc.cc:205] Tried to move back new op lower bound from 7333602614277570560 to 7333602612651040768. Current Snapshot: MvccSnapshot[applied={T|T < 7333602614277570560}]
I20260926 13:45:25.773209 21075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:25.769511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:44257 (local address 127.12.170.66:38737)
0926 13:45:25.769762 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:25.769774 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:25.769865 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:25.770065 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:25.770066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:25.770071 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:45:25.770174 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:25.770178 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.771122 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.771125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:25.772005 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:25.772014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.772598 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.772600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:25.772663 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:25.772701 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:45:25.772972 (+ 271us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:45:25.772976 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:45:25.773015 (+ 39us) client_negotiation.cc:240] Negotiation successful
0926 13:45:25.773072 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":77,"threads_started":1}
I20260926 13:45:25.773200 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:25.769572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:38737 (local address 127.12.170.67:44257)
0926 13:45:25.769681 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:25.769686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:25.769853 (+ 167us) server_negotiation.cc:407] Connection header received
0926 13:45:25.769932 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:25.769935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:25.769982 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:25.770063 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:45:25.770272 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.771034 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.772115 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.772477 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.772596 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:25.772818 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:45:25.772897 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:45:25.772975 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:25.773030 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 13:45:25.773078 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:45:25.794023 20496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:45:25.794183 20496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:45:25.796254 20496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e: Sending DeleteTablet for 3 replicas of tablet 0bfadc3f8869420598728ba25ab2d5b5
I20260926 13:45:25.796904 20634 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bfadc3f8869420598728ba25ab2d5b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:25 UTC) from {username='oryx', principal='oryx/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:50642
I20260926 13:45:25.797024 20774 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bfadc3f8869420598728ba25ab2d5b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:25 UTC) from {username='oryx', principal='oryx/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:43108
I20260926 13:45:25.797210 20914 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bfadc3f8869420598728ba25ab2d5b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:25 UTC) from {username='oryx', principal='oryx/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:60918
I20260926 13:45:25.797535 21081 tablet_replica.cc:330] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a: stopping tablet replica
I20260926 13:45:25.797657 21081 raft_consensus.cc:2229] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:45:25.797955 21081 raft_consensus.cc:2258] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:25.801628 21081 ts_tablet_manager.cc:1905] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:25.803790 21079 tablet_replica.cc:330] T 0bfadc3f8869420598728ba25ab2d5b5 P 0daec48270fb4665beb58e1cb1c672e3: stopping tablet replica
I20260926 13:45:25.803803 21080 tablet_replica.cc:330] T 0bfadc3f8869420598728ba25ab2d5b5 P 556c77c7adf74f3b83543fea8130ed35: stopping tablet replica
I20260926 13:45:25.803920 21079 raft_consensus.cc:2229] T 0bfadc3f8869420598728ba25ab2d5b5 P 0daec48270fb4665beb58e1cb1c672e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:45:25.803920 21080 raft_consensus.cc:2229] T 0bfadc3f8869420598728ba25ab2d5b5 P 556c77c7adf74f3b83543fea8130ed35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:45:25.804023 21079 raft_consensus.cc:2258] T 0bfadc3f8869420598728ba25ab2d5b5 P 0daec48270fb4665beb58e1cb1c672e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:25.804023 21080 raft_consensus.cc:2258] T 0bfadc3f8869420598728ba25ab2d5b5 P 556c77c7adf74f3b83543fea8130ed35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:25.804773 21080 ts_tablet_manager.cc:1905] T 0bfadc3f8869420598728ba25ab2d5b5 P 556c77c7adf74f3b83543fea8130ed35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:25.804809 21081 ts_tablet_manager.cc:1918] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:45:25.806496 21081 log.cc:1196] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/0bfadc3f8869420598728ba25ab2d5b5
I20260926 13:45:25.806815 21081 ts_tablet_manager.cc:1939] T 0bfadc3f8869420598728ba25ab2d5b5 P ff63674a2a8848d1a168dfbd580bbf3a: Deleting consensus metadata
I20260926 13:45:25.807240 21080 ts_tablet_manager.cc:1918] T 0bfadc3f8869420598728ba25ab2d5b5 P 556c77c7adf74f3b83543fea8130ed35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:45:25.807313 21080 log.cc:1196] T 0bfadc3f8869420598728ba25ab2d5b5 P 556c77c7adf74f3b83543fea8130ed35: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-1/wal/wals/0bfadc3f8869420598728ba25ab2d5b5
I20260926 13:45:25.807538 21080 ts_tablet_manager.cc:1939] T 0bfadc3f8869420598728ba25ab2d5b5 P 556c77c7adf74f3b83543fea8130ed35: Deleting consensus metadata
I20260926 13:45:25.807588 20481 catalog_manager.cc:4841] TS ff63674a2a8848d1a168dfbd580bbf3a (127.12.170.67:44257): tablet 0bfadc3f8869420598728ba25ab2d5b5 (table test-table [id=105cae234bfb4378ae3a0dd3b94e8559]) successfully deleted
I20260926 13:45:25.808135 20484 catalog_manager.cc:4841] TS 556c77c7adf74f3b83543fea8130ed35 (127.12.170.66:42103): tablet 0bfadc3f8869420598728ba25ab2d5b5 (table test-table [id=105cae234bfb4378ae3a0dd3b94e8559]) successfully deleted
I20260926 13:45:25.812016 21079 ts_tablet_manager.cc:1905] T 0bfadc3f8869420598728ba25ab2d5b5 P 0daec48270fb4665beb58e1cb1c672e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:25.814565 21079 ts_tablet_manager.cc:1918] T 0bfadc3f8869420598728ba25ab2d5b5 P 0daec48270fb4665beb58e1cb1c672e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:45:25.814641 21079 log.cc:1196] T 0bfadc3f8869420598728ba25ab2d5b5 P 0daec48270fb4665beb58e1cb1c672e3: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-0/wal/wals/0bfadc3f8869420598728ba25ab2d5b5
I20260926 13:45:25.814862 21079 ts_tablet_manager.cc:1939] T 0bfadc3f8869420598728ba25ab2d5b5 P 0daec48270fb4665beb58e1cb1c672e3: Deleting consensus metadata
I20260926 13:45:25.815562 20481 catalog_manager.cc:4841] TS 0daec48270fb4665beb58e1cb1c672e3 (127.12.170.65:45189): tablet 0bfadc3f8869420598728ba25ab2d5b5 (table test-table [id=105cae234bfb4378ae3a0dd3b94e8559]) successfully deleted
Sep 26 13:45:25 dist-test-slave-hc1k krb5kdc[20452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 13:45:25 dist-test-slave-hc1k krb5kdc[20452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:25 dist-test-slave-hc1k krb5kdc[20452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 13:45:25.829895 20546 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:25.823586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36634 (local address 127.12.170.126:43289)
0926 13:45:25.823725 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:25.823730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:25.823859 (+ 129us) server_negotiation.cc:407] Connection header received
0926 13:45:25.823984 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:25.823986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:25.824025 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:25.824088 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:25.824884 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.825393 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.826557 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.826794 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.826876 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:25.829775 (+ 2899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36634: BlockingRecv error: recv got EOF from 127.0.0.1:36634 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 13:45:25 dist-test-slave-hc1k krb5kdc[20452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM
I20260926 13:45:25.839440 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:25.831390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36636 (local address 127.12.170.126:43289)
0926 13:45:25.831484 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:25.831487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:25.831498 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:45:25.831525 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:25.831527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:25.831559 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:25.831617 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:25.832317 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.832829 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.833906 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.834108 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.834190 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:25.836539 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:25.836542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:25.836543 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:25.836566 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:25.837902 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:25.838301 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:25.838305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:25.838307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:25.838384 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:25.838637 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:25.838641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:25.838643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:25.838824 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:25.838955 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:25.839174 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 13:45:25.839281 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 13:45:25.841894 20496 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36636:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:45:25.842346 20496 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:45:25.846599 20634 tablet_service.cc:1467] Processing CreateTablet for tablet 8add78f60db94f31b76a9b0522c98546 (DEFAULT_TABLE table=test-table [id=8404f67726ca4097909d6361d7452949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:25.846801 20774 tablet_service.cc:1467] Processing CreateTablet for tablet 8add78f60db94f31b76a9b0522c98546 (DEFAULT_TABLE table=test-table [id=8404f67726ca4097909d6361d7452949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:25.846866 20634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8add78f60db94f31b76a9b0522c98546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:25.847041 20774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8add78f60db94f31b76a9b0522c98546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:25.848984 21010 tablet_bootstrap.cc:492] T 8add78f60db94f31b76a9b0522c98546 P 0daec48270fb4665beb58e1cb1c672e3: Bootstrap starting.
I20260926 13:45:25.849157 20914 tablet_service.cc:1467] Processing CreateTablet for tablet 8add78f60db94f31b76a9b0522c98546 (DEFAULT_TABLE table=test-table [id=8404f67726ca4097909d6361d7452949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:25.849255 21008 tablet_bootstrap.cc:492] T 8add78f60db94f31b76a9b0522c98546 P 556c77c7adf74f3b83543fea8130ed35: Bootstrap starting.
I20260926 13:45:25.849406 20914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8add78f60db94f31b76a9b0522c98546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:25.849771 21010 tablet_bootstrap.cc:654] T 8add78f60db94f31b76a9b0522c98546 P 0daec48270fb4665beb58e1cb1c672e3: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:25.850005 21008 tablet_bootstrap.cc:654] T 8add78f60db94f31b76a9b0522c98546 P 556c77c7adf74f3b83543fea8130ed35: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:25.851150 21008 tablet_bootstrap.cc:492] T 8add78f60db94f31b76a9b0522c98546 P 556c77c7adf74f3b83543fea8130ed35: No bootstrap required, opened a new log
I20260926 13:45:25.851214 21008 ts_tablet_manager.cc:1397] T 8add78f60db94f31b76a9b0522c98546 P 556c77c7adf74f3b83543fea8130ed35: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:25.851531 21009 tablet_bootstrap.cc:492] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a: Bootstrap starting.
I20260926 13:45:25.851584 21008 raft_consensus.cc:348] T 8add78f60db94f31b76a9b0522c98546 P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.851715 21008 raft_consensus.cc:374] T 8add78f60db94f31b76a9b0522c98546 P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:25.851752 21008 raft_consensus.cc:729] T 8add78f60db94f31b76a9b0522c98546 P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 556c77c7adf74f3b83543fea8130ed35, State: Initialized, Role: FOLLOWER
I20260926 13:45:25.851864 21008 consensus_queue.cc:260] T 8add78f60db94f31b76a9b0522c98546 P 556c77c7adf74f3b83543fea8130ed35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.852099 21008 ts_tablet_manager.cc:1428] T 8add78f60db94f31b76a9b0522c98546 P 556c77c7adf74f3b83543fea8130ed35: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:45:25.852350 21009 tablet_bootstrap.cc:654] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:25.853354 21010 tablet_bootstrap.cc:492] T 8add78f60db94f31b76a9b0522c98546 P 0daec48270fb4665beb58e1cb1c672e3: No bootstrap required, opened a new log
I20260926 13:45:25.853430 21010 ts_tablet_manager.cc:1397] T 8add78f60db94f31b76a9b0522c98546 P 0daec48270fb4665beb58e1cb1c672e3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 13:45:25.853749 21010 raft_consensus.cc:348] T 8add78f60db94f31b76a9b0522c98546 P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.853849 21010 raft_consensus.cc:374] T 8add78f60db94f31b76a9b0522c98546 P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:25.853886 21010 raft_consensus.cc:729] T 8add78f60db94f31b76a9b0522c98546 P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0daec48270fb4665beb58e1cb1c672e3, State: Initialized, Role: FOLLOWER
I20260926 13:45:25.853996 21010 consensus_queue.cc:260] T 8add78f60db94f31b76a9b0522c98546 P 0daec48270fb4665beb58e1cb1c672e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.854233 21010 ts_tablet_manager.cc:1428] T 8add78f60db94f31b76a9b0522c98546 P 0daec48270fb4665beb58e1cb1c672e3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:45:25.854970 21009 tablet_bootstrap.cc:492] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a: No bootstrap required, opened a new log
I20260926 13:45:25.855042 21009 ts_tablet_manager.cc:1397] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 13:45:25.855381 21009 raft_consensus.cc:348] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.855466 21009 raft_consensus.cc:374] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:25.855495 21009 raft_consensus.cc:729] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff63674a2a8848d1a168dfbd580bbf3a, State: Initialized, Role: FOLLOWER
I20260926 13:45:25.855604 21009 consensus_queue.cc:260] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.855849 21009 ts_tablet_manager.cc:1428] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:45:25.857319 21016 raft_consensus.cc:482] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:45:25.857398 21016 raft_consensus.cc:504] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.857726 21016 leader_election.cc:288] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0daec48270fb4665beb58e1cb1c672e3 (127.12.170.65:45189), 556c77c7adf74f3b83543fea8130ed35 (127.12.170.66:42103)
I20260926 13:45:25.858021 20654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8add78f60db94f31b76a9b0522c98546" candidate_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0daec48270fb4665beb58e1cb1c672e3" is_pre_election: true
I20260926 13:45:25.858074 20794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8add78f60db94f31b76a9b0522c98546" candidate_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "556c77c7adf74f3b83543fea8130ed35" is_pre_election: true
I20260926 13:45:25.858135 20654 raft_consensus.cc:2454] T 8add78f60db94f31b76a9b0522c98546 P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff63674a2a8848d1a168dfbd580bbf3a in term 0.
I20260926 13:45:25.858178 20794 raft_consensus.cc:2454] T 8add78f60db94f31b76a9b0522c98546 P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff63674a2a8848d1a168dfbd580bbf3a in term 0.
I20260926 13:45:25.858403 20865 leader_election.cc:302] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0daec48270fb4665beb58e1cb1c672e3, ff63674a2a8848d1a168dfbd580bbf3a; no voters:
I20260926 13:45:25.858639 21016 raft_consensus.cc:2781] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:45:25.858703 21016 raft_consensus.cc:482] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:45:25.858747 21016 raft_consensus.cc:3037] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:25.859675 21016 raft_consensus.cc:504] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.859985 21016 leader_election.cc:288] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a [CANDIDATE]: Term 1 election: Requested vote from peers 0daec48270fb4665beb58e1cb1c672e3 (127.12.170.65:45189), 556c77c7adf74f3b83543fea8130ed35 (127.12.170.66:42103)
I20260926 13:45:25.860221 20654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8add78f60db94f31b76a9b0522c98546" candidate_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0daec48270fb4665beb58e1cb1c672e3"
I20260926 13:45:25.860337 20654 raft_consensus.cc:3037] T 8add78f60db94f31b76a9b0522c98546 P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:25.860420 20794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8add78f60db94f31b76a9b0522c98546" candidate_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "556c77c7adf74f3b83543fea8130ed35"
I20260926 13:45:25.860513 20794 raft_consensus.cc:3037] T 8add78f60db94f31b76a9b0522c98546 P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:25.861186 20654 raft_consensus.cc:2454] T 8add78f60db94f31b76a9b0522c98546 P 0daec48270fb4665beb58e1cb1c672e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff63674a2a8848d1a168dfbd580bbf3a in term 1.
I20260926 13:45:25.861421 20794 raft_consensus.cc:2454] T 8add78f60db94f31b76a9b0522c98546 P 556c77c7adf74f3b83543fea8130ed35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff63674a2a8848d1a168dfbd580bbf3a in term 1.
I20260926 13:45:25.861467 20865 leader_election.cc:302] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0daec48270fb4665beb58e1cb1c672e3, ff63674a2a8848d1a168dfbd580bbf3a; no voters:
I20260926 13:45:25.861621 21016 raft_consensus.cc:2781] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:25.861701 21016 raft_consensus.cc:686] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a [term 1 LEADER]: Becoming Leader. State: Replica: ff63674a2a8848d1a168dfbd580bbf3a, State: Running, Role: LEADER
I20260926 13:45:25.861804 21016 consensus_queue.cc:237] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.863243 20495 catalog_manager.cc:5495] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a reported cstate change: term changed from 0 to 1, leader changed from <none> to ff63674a2a8848d1a168dfbd580bbf3a (127.12.170.67). New cstate: current_term: 1 leader_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } health_report { overall_health: HEALTHY } } }
I20260926 13:45:25.885022 21052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:25.881514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60924 (local address 127.12.170.67:44257)
0926 13:45:25.881624 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:25.881629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:25.881643 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:45:25.881692 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:25.881694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:25.881738 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:25.881808 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:45:25.882026 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.882782 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.883967 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:25.884320 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:25.884413 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:25.884645 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:45:25.884721 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:45:25.884774 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:25.884851 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 13:45:25.884895 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:45:25.886780 20794 raft_consensus.cc:1261] T 8add78f60db94f31b76a9b0522c98546 P 556c77c7adf74f3b83543fea8130ed35 [term 1 FOLLOWER]: Refusing update from remote peer ff63674a2a8848d1a168dfbd580bbf3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:25.886968 20654 raft_consensus.cc:1261] T 8add78f60db94f31b76a9b0522c98546 P 0daec48270fb4665beb58e1cb1c672e3 [term 1 FOLLOWER]: Refusing update from remote peer ff63674a2a8848d1a168dfbd580bbf3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:25.887179 21039 consensus_queue.cc:1035] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:25.887379 21016 consensus_queue.cc:1035] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:25.893307 20495 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:45:25.893417 20495 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:45:25.895157 20495 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e: Sending DeleteTablet for 3 replicas of tablet 8add78f60db94f31b76a9b0522c98546
I20260926 13:45:25.895690 20634 tablet_service.cc:1514] Processing DeleteTablet for tablet 8add78f60db94f31b76a9b0522c98546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:25 UTC) from {username='oryx', principal='oryx/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:50642
I20260926 13:45:25.895690 20774 tablet_service.cc:1514] Processing DeleteTablet for tablet 8add78f60db94f31b76a9b0522c98546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:25 UTC) from {username='oryx', principal='oryx/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:43108
I20260926 13:45:25.895772 20914 tablet_service.cc:1514] Processing DeleteTablet for tablet 8add78f60db94f31b76a9b0522c98546 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:25 UTC) from {username='oryx', principal='oryx/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:60918
I20260926 13:45:25.895851 21080 tablet_replica.cc:330] T 8add78f60db94f31b76a9b0522c98546 P 556c77c7adf74f3b83543fea8130ed35: stopping tablet replica
I20260926 13:45:25.895937 21081 tablet_replica.cc:330] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a: stopping tablet replica
I20260926 13:45:25.895941 21079 tablet_replica.cc:330] T 8add78f60db94f31b76a9b0522c98546 P 0daec48270fb4665beb58e1cb1c672e3: stopping tablet replica
I20260926 13:45:25.895957 21080 raft_consensus.cc:2229] T 8add78f60db94f31b76a9b0522c98546 P 556c77c7adf74f3b83543fea8130ed35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:45:25.896024 21081 raft_consensus.cc:2229] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:45:25.896042 21080 pending_rounds.cc:62] T 8add78f60db94f31b76a9b0522c98546 P 556c77c7adf74f3b83543fea8130ed35: Trying to abort 2 pending ops.
I20260926 13:45:25.896050 21079 raft_consensus.cc:2229] T 8add78f60db94f31b76a9b0522c98546 P 0daec48270fb4665beb58e1cb1c672e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:45:25.896152 21081 raft_consensus.cc:2258] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:25.896159 21079 raft_consensus.cc:2258] T 8add78f60db94f31b76a9b0522c98546 P 0daec48270fb4665beb58e1cb1c672e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:25.896092 21080 pending_rounds.cc:68] T 8add78f60db94f31b76a9b0522c98546 P 556c77c7adf74f3b83543fea8130ed35: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7333602614828494848 op_type: WRITE_OP write_request { tablet_id: "8add78f60db94f31b76a9b0522c98546" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\241\233\337\325\006\"8\n\ntest-admin\022*\n 8404f67726ca4097909d6361d7452949\020\001\030\001 \001(\001" signature: "<redacted>""}\217\313\255\014\336\034\r\212\037\334\365\036dC\237\363V\265\327\227\223`\311\273%I\252\353\362X\240/kS2%\247[*b\n\2063\373\r\322\200\200~\2277\252\214\214\265\306-|\326\232\010\254\330" signing_key_seq_num: 0 } } request_id { client_id: "dd577f174bf4436c92b7baaf246dc104" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
W20260926 13:45:25.896229 21100 consensus_queue.cc:1159] T 8add78f60db94f31b76a9b0522c98546 P 556c77c7adf74f3b83543fea8130ed35 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "556c77c7adf74f3b83543fea8130ed35" status { last_received { term: 1 index: 2 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 2 } }
I20260926 13:45:25.896436 21080 pending_rounds.cc:68] T 8add78f60db94f31b76a9b0522c98546 P 556c77c7adf74f3b83543fea8130ed35: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7333602614731223040 op_type: NO_OP noop_request { }
I20260926 13:45:25.896505 21080 raft_consensus.cc:2866] T 8add78f60db94f31b76a9b0522c98546 P 556c77c7adf74f3b83543fea8130ed35 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260926 13:45:25.896561 21080 raft_consensus.cc:2258] T 8add78f60db94f31b76a9b0522c98546 P 556c77c7adf74f3b83543fea8130ed35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:25.897135 21081 ts_tablet_manager.cc:1905] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:25.897136 21079 ts_tablet_manager.cc:1905] T 8add78f60db94f31b76a9b0522c98546 P 0daec48270fb4665beb58e1cb1c672e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:25.899958 21080 ts_tablet_manager.cc:1905] T 8add78f60db94f31b76a9b0522c98546 P 556c77c7adf74f3b83543fea8130ed35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:25.900386 20495 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36636:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:45:25.900746 20495 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:45:25.901522 21079 ts_tablet_manager.cc:1918] T 8add78f60db94f31b76a9b0522c98546 P 0daec48270fb4665beb58e1cb1c672e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:45:25.901597 21079 log.cc:1196] T 8add78f60db94f31b76a9b0522c98546 P 0daec48270fb4665beb58e1cb1c672e3: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-0/wal/wals/8add78f60db94f31b76a9b0522c98546
I20260926 13:45:25.901825 21079 ts_tablet_manager.cc:1939] T 8add78f60db94f31b76a9b0522c98546 P 0daec48270fb4665beb58e1cb1c672e3: Deleting consensus metadata
I20260926 13:45:25.902355 21080 ts_tablet_manager.cc:1918] T 8add78f60db94f31b76a9b0522c98546 P 556c77c7adf74f3b83543fea8130ed35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:45:25.902426 21080 log.cc:1196] T 8add78f60db94f31b76a9b0522c98546 P 556c77c7adf74f3b83543fea8130ed35: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-1/wal/wals/8add78f60db94f31b76a9b0522c98546
I20260926 13:45:25.902477 20481 catalog_manager.cc:4841] TS 0daec48270fb4665beb58e1cb1c672e3 (127.12.170.65:45189): tablet 8add78f60db94f31b76a9b0522c98546 (table test-table [id=8404f67726ca4097909d6361d7452949]) successfully deleted
I20260926 13:45:25.902634 21080 ts_tablet_manager.cc:1939] T 8add78f60db94f31b76a9b0522c98546 P 556c77c7adf74f3b83543fea8130ed35: Deleting consensus metadata
I20260926 13:45:25.903280 21081 ts_tablet_manager.cc:1918] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:45:25.903302 20484 catalog_manager.cc:4841] TS 556c77c7adf74f3b83543fea8130ed35 (127.12.170.66:42103): tablet 8add78f60db94f31b76a9b0522c98546 (table test-table [id=8404f67726ca4097909d6361d7452949]) successfully deleted
I20260926 13:45:25.903357 21081 log.cc:1196] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/8add78f60db94f31b76a9b0522c98546
I20260926 13:45:25.903568 21081 ts_tablet_manager.cc:1939] T 8add78f60db94f31b76a9b0522c98546 P ff63674a2a8848d1a168dfbd580bbf3a: Deleting consensus metadata
I20260926 13:45:25.904479 20481 catalog_manager.cc:4841] TS ff63674a2a8848d1a168dfbd580bbf3a (127.12.170.67:44257): tablet 8add78f60db94f31b76a9b0522c98546 (table test-table [id=8404f67726ca4097909d6361d7452949]) successfully deleted
I20260926 13:45:25.905598 20634 tablet_service.cc:1467] Processing CreateTablet for tablet 41ae5e7ecb894bf18492d9933a52e377 (DEFAULT_TABLE table=test-table [id=62a8ff47449b4026aae8e43a011c4e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:25.905829 20634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ae5e7ecb894bf18492d9933a52e377. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:25.905835 20774 tablet_service.cc:1467] Processing CreateTablet for tablet 41ae5e7ecb894bf18492d9933a52e377 (DEFAULT_TABLE table=test-table [id=62a8ff47449b4026aae8e43a011c4e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:25.905869 20914 tablet_service.cc:1467] Processing CreateTablet for tablet 41ae5e7ecb894bf18492d9933a52e377 (DEFAULT_TABLE table=test-table [id=62a8ff47449b4026aae8e43a011c4e53]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:25.906026 20774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ae5e7ecb894bf18492d9933a52e377. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:25.906098 20914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ae5e7ecb894bf18492d9933a52e377. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:25.907968 21010 tablet_bootstrap.cc:492] T 41ae5e7ecb894bf18492d9933a52e377 P 0daec48270fb4665beb58e1cb1c672e3: Bootstrap starting.
I20260926 13:45:25.908216 21009 tablet_bootstrap.cc:492] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a: Bootstrap starting.
I20260926 13:45:25.908281 21008 tablet_bootstrap.cc:492] T 41ae5e7ecb894bf18492d9933a52e377 P 556c77c7adf74f3b83543fea8130ed35: Bootstrap starting.
I20260926 13:45:25.908743 21010 tablet_bootstrap.cc:654] T 41ae5e7ecb894bf18492d9933a52e377 P 0daec48270fb4665beb58e1cb1c672e3: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:25.909024 21009 tablet_bootstrap.cc:654] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:25.909040 21008 tablet_bootstrap.cc:654] T 41ae5e7ecb894bf18492d9933a52e377 P 556c77c7adf74f3b83543fea8130ed35: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:25.910166 21009 tablet_bootstrap.cc:492] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a: No bootstrap required, opened a new log
I20260926 13:45:25.910243 21009 ts_tablet_manager.cc:1397] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:25.910333 21008 tablet_bootstrap.cc:492] T 41ae5e7ecb894bf18492d9933a52e377 P 556c77c7adf74f3b83543fea8130ed35: No bootstrap required, opened a new log
I20260926 13:45:25.910404 21008 ts_tablet_manager.cc:1397] T 41ae5e7ecb894bf18492d9933a52e377 P 556c77c7adf74f3b83543fea8130ed35: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:25.910627 21010 tablet_bootstrap.cc:492] T 41ae5e7ecb894bf18492d9933a52e377 P 0daec48270fb4665beb58e1cb1c672e3: No bootstrap required, opened a new log
I20260926 13:45:25.910699 21010 ts_tablet_manager.cc:1397] T 41ae5e7ecb894bf18492d9933a52e377 P 0daec48270fb4665beb58e1cb1c672e3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 13:45:25.910656 21009 raft_consensus.cc:348] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.910763 21009 raft_consensus.cc:374] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:25.910795 21009 raft_consensus.cc:729] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff63674a2a8848d1a168dfbd580bbf3a, State: Initialized, Role: FOLLOWER
I20260926 13:45:25.910742 21008 raft_consensus.cc:348] T 41ae5e7ecb894bf18492d9933a52e377 P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.910837 21008 raft_consensus.cc:374] T 41ae5e7ecb894bf18492d9933a52e377 P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:25.910867 21008 raft_consensus.cc:729] T 41ae5e7ecb894bf18492d9933a52e377 P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 556c77c7adf74f3b83543fea8130ed35, State: Initialized, Role: FOLLOWER
I20260926 13:45:25.910905 21009 consensus_queue.cc:260] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.910975 21008 consensus_queue.cc:260] T 41ae5e7ecb894bf18492d9933a52e377 P 556c77c7adf74f3b83543fea8130ed35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.911041 21010 raft_consensus.cc:348] T 41ae5e7ecb894bf18492d9933a52e377 P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.911127 21010 raft_consensus.cc:374] T 41ae5e7ecb894bf18492d9933a52e377 P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:25.911154 21009 ts_tablet_manager.cc:1428] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 13:45:25.911156 21010 raft_consensus.cc:729] T 41ae5e7ecb894bf18492d9933a52e377 P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0daec48270fb4665beb58e1cb1c672e3, State: Initialized, Role: FOLLOWER
I20260926 13:45:25.911213 21008 ts_tablet_manager.cc:1428] T 41ae5e7ecb894bf18492d9933a52e377 P 556c77c7adf74f3b83543fea8130ed35: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:45:25.911273 21010 consensus_queue.cc:260] T 41ae5e7ecb894bf18492d9933a52e377 P 0daec48270fb4665beb58e1cb1c672e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.911510 21010 ts_tablet_manager.cc:1428] T 41ae5e7ecb894bf18492d9933a52e377 P 0daec48270fb4665beb58e1cb1c672e3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:45:25.994832 21101 raft_consensus.cc:482] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:45:25.994972 21101 raft_consensus.cc:504] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.995358 21101 leader_election.cc:288] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0daec48270fb4665beb58e1cb1c672e3 (127.12.170.65:45189), 556c77c7adf74f3b83543fea8130ed35 (127.12.170.66:42103)
I20260926 13:45:25.995770 20654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ae5e7ecb894bf18492d9933a52e377" candidate_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0daec48270fb4665beb58e1cb1c672e3" is_pre_election: true
I20260926 13:45:25.995924 20654 raft_consensus.cc:2454] T 41ae5e7ecb894bf18492d9933a52e377 P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff63674a2a8848d1a168dfbd580bbf3a in term 0.
I20260926 13:45:25.995908 20794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ae5e7ecb894bf18492d9933a52e377" candidate_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "556c77c7adf74f3b83543fea8130ed35" is_pre_election: true
I20260926 13:45:25.996057 20794 raft_consensus.cc:2454] T 41ae5e7ecb894bf18492d9933a52e377 P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff63674a2a8848d1a168dfbd580bbf3a in term 0.
I20260926 13:45:25.996201 20865 leader_election.cc:302] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0daec48270fb4665beb58e1cb1c672e3, ff63674a2a8848d1a168dfbd580bbf3a; no voters:
I20260926 13:45:25.996358 21101 raft_consensus.cc:2781] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:45:25.996415 21101 raft_consensus.cc:482] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:45:25.996460 21101 raft_consensus.cc:3037] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:25.997532 21101 raft_consensus.cc:504] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:25.997851 21101 leader_election.cc:288] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a [CANDIDATE]: Term 1 election: Requested vote from peers 0daec48270fb4665beb58e1cb1c672e3 (127.12.170.65:45189), 556c77c7adf74f3b83543fea8130ed35 (127.12.170.66:42103)
I20260926 13:45:25.998179 20794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ae5e7ecb894bf18492d9933a52e377" candidate_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "556c77c7adf74f3b83543fea8130ed35"
I20260926 13:45:25.998180 20654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ae5e7ecb894bf18492d9933a52e377" candidate_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0daec48270fb4665beb58e1cb1c672e3"
I20260926 13:45:25.998281 20654 raft_consensus.cc:3037] T 41ae5e7ecb894bf18492d9933a52e377 P 0daec48270fb4665beb58e1cb1c672e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:25.998292 20794 raft_consensus.cc:3037] T 41ae5e7ecb894bf18492d9933a52e377 P 556c77c7adf74f3b83543fea8130ed35 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:25.999110 20654 raft_consensus.cc:2454] T 41ae5e7ecb894bf18492d9933a52e377 P 0daec48270fb4665beb58e1cb1c672e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff63674a2a8848d1a168dfbd580bbf3a in term 1.
I20260926 13:45:25.999168 20794 raft_consensus.cc:2454] T 41ae5e7ecb894bf18492d9933a52e377 P 556c77c7adf74f3b83543fea8130ed35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff63674a2a8848d1a168dfbd580bbf3a in term 1.
I20260926 13:45:25.999347 20865 leader_election.cc:302] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0daec48270fb4665beb58e1cb1c672e3, ff63674a2a8848d1a168dfbd580bbf3a; no voters:
I20260926 13:45:25.999485 21101 raft_consensus.cc:2781] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:25.999565 21101 raft_consensus.cc:686] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a [term 1 LEADER]: Becoming Leader. State: Replica: ff63674a2a8848d1a168dfbd580bbf3a, State: Running, Role: LEADER
I20260926 13:45:25.999723 21101 consensus_queue.cc:237] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } }
I20260926 13:45:26.001323 20497 catalog_manager.cc:5495] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a reported cstate change: term changed from 0 to 1, leader changed from <none> to ff63674a2a8848d1a168dfbd580bbf3a (127.12.170.67). New cstate: current_term: 1 leader_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff63674a2a8848d1a168dfbd580bbf3a" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44257 } health_report { overall_health: HEALTHY } } }
I20260926 13:45:26.016594 20794 raft_consensus.cc:1261] T 41ae5e7ecb894bf18492d9933a52e377 P 556c77c7adf74f3b83543fea8130ed35 [term 1 FOLLOWER]: Refusing update from remote peer ff63674a2a8848d1a168dfbd580bbf3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:26.016594 20653 raft_consensus.cc:1261] T 41ae5e7ecb894bf18492d9933a52e377 P 0daec48270fb4665beb58e1cb1c672e3 [term 1 FOLLOWER]: Refusing update from remote peer ff63674a2a8848d1a168dfbd580bbf3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:26.016956 21016 consensus_queue.cc:1035] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0daec48270fb4665beb58e1cb1c672e3" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 45189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:26.017093 21101 consensus_queue.cc:1035] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "556c77c7adf74f3b83543fea8130ed35" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:26.044533 20497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:45:26.044663 20497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:45:26.046253 20497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b7c2d92ec29421aad8eee9cfe59c98e: Sending DeleteTablet for 3 replicas of tablet 41ae5e7ecb894bf18492d9933a52e377
I20260926 13:45:26.046697 20634 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ae5e7ecb894bf18492d9933a52e377 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:26 UTC) from {username='oryx', principal='oryx/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:50642
I20260926 13:45:26.046741 20914 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ae5e7ecb894bf18492d9933a52e377 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:26 UTC) from {username='oryx', principal='oryx/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:60918
I20260926 13:45:26.046868 21079 tablet_replica.cc:330] T 41ae5e7ecb894bf18492d9933a52e377 P 0daec48270fb4665beb58e1cb1c672e3: stopping tablet replica
I20260926 13:45:26.046893 21081 tablet_replica.cc:330] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a: stopping tablet replica
I20260926 13:45:26.046928 20774 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ae5e7ecb894bf18492d9933a52e377 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:26 UTC) from {username='oryx', principal='oryx/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:43108
I20260926 13:45:26.046986 21081 raft_consensus.cc:2229] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:45:26.046986 21079 raft_consensus.cc:2229] T 41ae5e7ecb894bf18492d9933a52e377 P 0daec48270fb4665beb58e1cb1c672e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:45:26.047080 21080 tablet_replica.cc:330] T 41ae5e7ecb894bf18492d9933a52e377 P 556c77c7adf74f3b83543fea8130ed35: stopping tablet replica
I20260926 13:45:26.047077 21079 raft_consensus.cc:2258] T 41ae5e7ecb894bf18492d9933a52e377 P 0daec48270fb4665beb58e1cb1c672e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:26.047132 21081 raft_consensus.cc:2258] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:26.047184 21080 raft_consensus.cc:2229] T 41ae5e7ecb894bf18492d9933a52e377 P 556c77c7adf74f3b83543fea8130ed35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:45:26.047271 21080 raft_consensus.cc:2258] T 41ae5e7ecb894bf18492d9933a52e377 P 556c77c7adf74f3b83543fea8130ed35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:26.047789 21081 ts_tablet_manager.cc:1905] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:26.047834 21080 ts_tablet_manager.cc:1905] T 41ae5e7ecb894bf18492d9933a52e377 P 556c77c7adf74f3b83543fea8130ed35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:26.047869 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 20569
I20260926 13:45:26.048008 21079 ts_tablet_manager.cc:1905] T 41ae5e7ecb894bf18492d9933a52e377 P 0daec48270fb4665beb58e1cb1c672e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:26.049813 21081 ts_tablet_manager.cc:1918] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:45:26.049925 21080 ts_tablet_manager.cc:1918] T 41ae5e7ecb894bf18492d9933a52e377 P 556c77c7adf74f3b83543fea8130ed35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:45:26.049974 21080 log.cc:1196] T 41ae5e7ecb894bf18492d9933a52e377 P 556c77c7adf74f3b83543fea8130ed35: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-1/wal/wals/41ae5e7ecb894bf18492d9933a52e377
I20260926 13:45:26.049978 21081 log.cc:1196] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/41ae5e7ecb894bf18492d9933a52e377
I20260926 13:45:26.050150 21080 ts_tablet_manager.cc:1939] T 41ae5e7ecb894bf18492d9933a52e377 P 556c77c7adf74f3b83543fea8130ed35: Deleting consensus metadata
I20260926 13:45:26.050194 21081 ts_tablet_manager.cc:1939] T 41ae5e7ecb894bf18492d9933a52e377 P ff63674a2a8848d1a168dfbd580bbf3a: Deleting consensus metadata
I20260926 13:45:26.050750 20484 catalog_manager.cc:4841] TS 556c77c7adf74f3b83543fea8130ed35 (127.12.170.66:42103): tablet 41ae5e7ecb894bf18492d9933a52e377 (table test-table [id=62a8ff47449b4026aae8e43a011c4e53]) successfully deleted
I20260926 13:45:26.050836 20481 catalog_manager.cc:4841] TS ff63674a2a8848d1a168dfbd580bbf3a (127.12.170.67:44257): tablet 41ae5e7ecb894bf18492d9933a52e377 (table test-table [id=62a8ff47449b4026aae8e43a011c4e53]) successfully deleted
W20260926 13:45:26.056183 20481 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.170.65:45189 (error 108)
I20260926 13:45:26.056243 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 20709
I20260926 13:45:26.058076 21109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:26.057204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:45189 (local address 127.0.0.1:50650)
0926 13:45:26.057926 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:26.057989 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.65:45189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":58,"threads_started":1}
W20260926 13:45:26.058182 20481 catalog_manager.cc:4584] TS 0daec48270fb4665beb58e1cb1c672e3 (127.12.170.65:45189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 41ae5e7ecb894bf18492d9933a52e377: Network error: Client connection negotiation failed: client connection to 127.12.170.65:45189: connect: Connection refused (error 111)
I20260926 13:45:26.062196 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 20849
I20260926 13:45:26.067306 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 20465
2026-09-26T13:45:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3071 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:45:26 dist-test-slave-hc1k krb5kdc[21112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:45:26 dist-test-slave-hc1k krb5kdc[21112](info): set up 2 sockets
Sep 26 13:45:26 dist-test-slave-hc1k krb5kdc[21112](info): commencing operation
krb5kdc: starting...
W20260926 13:45:27.008193 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:45:27 dist-test-slave-hc1k krb5kdc[21112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:45:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:45:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:45:27.114078 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:40611
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:43889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:40611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:27.217267 21125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:27.217504 21125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:27.217557 21125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:27.221423 21125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:45:27.221493 21125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:27.221519 21125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:27.221546 21125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:45:27.221576 21125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:45:27.226585 21125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:43889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:40611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:40611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:27.227561 21125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:27.228506 21125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:27.236212 21133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:27.236248 21130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:27.236316 21125 server_base.cc:1034] running on GCE node
W20260926 13:45:27.236264 21131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:27.236899 21125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:27.237847 21125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:27.239003 21125 hybrid_clock.cc:648] HybridClock initialized: now 1790430327238992 us; error 24 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T13:45:27Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1184 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:45:27 dist-test-slave-hc1k krb5kdc[21138](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:45:27 dist-test-slave-hc1k krb5kdc[21138](info): set up 2 sockets
Sep 26 13:45:27 dist-test-slave-hc1k krb5kdc[21138](info): commencing operation
krb5kdc: starting...
W20260926 13:45:28.191952 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:45:28 dist-test-slave-hc1k krb5kdc[21138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:45:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:45:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:45:28.300673 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:40817
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:39721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:40817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:28.402899 21151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:28.403167 21151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:28.403228 21151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:28.407198 21151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:45:28.407264 21151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:28.407290 21151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:28.407310 21151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:45:28.407330 21151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:45:28.412446 21151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:39721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:40817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:40817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:28.413424 21151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:28.414394 21151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:28.422092 21159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:28.422184 21157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:28.422302 21151 server_base.cc:1034] running on GCE node
W20260926 13:45:28.422101 21156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:28.422888 21151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:28.423906 21151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:28.425050 21151 hybrid_clock.cc:648] HybridClock initialized: now 1790430328425035 us; error 27 us; skew 500 ppm
Sep 26 13:45:28 dist-test-slave-hc1k krb5kdc[21138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430328, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:28.430747 21151 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:45:28.431535 21151 webserver.cc:460] Webserver started at http://127.12.170.126:42973/ using document root <none> and password file <none>
I20260926 13:45:28.432078 21151 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:28.432142 21151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:28.432320 21151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:28.434227 21151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "815f6b54c3ef4096823eda6e73eb0efd"
format_stamp: "Formatted at 2026-09-26 13:45:28 on dist-test-slave-hc1k"
server_key: "b0ee49eb003808b91e0840776c65c656"
server_key_iv: "886a34bdbcfe31396012dfc1f9c02657"
server_key_version: "encryptionkey@0"
I20260926 13:45:28.434780 21151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "815f6b54c3ef4096823eda6e73eb0efd"
format_stamp: "Formatted at 2026-09-26 13:45:28 on dist-test-slave-hc1k"
server_key: "b0ee49eb003808b91e0840776c65c656"
server_key_iv: "886a34bdbcfe31396012dfc1f9c02657"
server_key_version: "encryptionkey@0"
I20260926 13:45:28.438174 21151 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 13:45:28.440380 21166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:28.441507 21151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:28.441617 21151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "815f6b54c3ef4096823eda6e73eb0efd"
format_stamp: "Formatted at 2026-09-26 13:45:28 on dist-test-slave-hc1k"
server_key: "b0ee49eb003808b91e0840776c65c656"
server_key_iv: "886a34bdbcfe31396012dfc1f9c02657"
server_key_version: "encryptionkey@0"
I20260926 13:45:28.441702 21151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:28.466543 21151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:28.469487 21151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:28.469656 21151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:28.476323 21151 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:40817
I20260926 13:45:28.476379 21218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:40817 every 8 connection(s)
I20260926 13:45:28.477166 21151 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:45:28.479745 21219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:28.484853 21219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd: Bootstrap starting.
I20260926 13:45:28.485443 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 21151
I20260926 13:45:28.485560 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:45:28.485839 12969 external_mini_cluster.cc:1428] Setting key 9ac463c12a12229334226a5d464fec7c
I20260926 13:45:28.487493 21219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:28.488189 21219 log.cc:824] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:28.490087 21219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd: No bootstrap required, opened a new log
I20260926 13:45:28.493820 21219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815f6b54c3ef4096823eda6e73eb0efd" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40817 } }
I20260926 13:45:28.494020 21219 raft_consensus.cc:374] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:28.494072 21219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 815f6b54c3ef4096823eda6e73eb0efd, State: Initialized, Role: FOLLOWER
Sep 26 13:45:28 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:28.494477 21219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815f6b54c3ef4096823eda6e73eb0efd" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40817 } }
I20260926 13:45:28.494601 21219 raft_consensus.cc:388] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:45:28.494663 21219 raft_consensus.cc:482] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:45:28.494766 21219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:28.495692 21219 raft_consensus.cc:504] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815f6b54c3ef4096823eda6e73eb0efd" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40817 } }
I20260926 13:45:28.496043 21219 leader_election.cc:302] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 815f6b54c3ef4096823eda6e73eb0efd; no voters:
I20260926 13:45:28.496325 21219 leader_election.cc:288] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:45:28.496423 21224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:28.496608 21224 raft_consensus.cc:686] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [term 1 LEADER]: Becoming Leader. State: Replica: 815f6b54c3ef4096823eda6e73eb0efd, State: Running, Role: LEADER
I20260926 13:45:28.496922 21224 consensus_queue.cc:237] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815f6b54c3ef4096823eda6e73eb0efd" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40817 } }
I20260926 13:45:28.497386 21219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:45:28.498097 21224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 815f6b54c3ef4096823eda6e73eb0efd. Latest consensus state: current_term: 1 leader_uuid: "815f6b54c3ef4096823eda6e73eb0efd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815f6b54c3ef4096823eda6e73eb0efd" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40817 } } }
I20260926 13:45:28.498208 21224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:28.498497 21225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "815f6b54c3ef4096823eda6e73eb0efd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815f6b54c3ef4096823eda6e73eb0efd" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40817 } } }
I20260926 13:45:28.498580 21225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:28.498661 21233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:45:28.499984 21222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:28.487013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49754 (local address 127.12.170.126:40817)
0926 13:45:28.487397 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:28.487412 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:28.487446 (+ 34us) server_negotiation.cc:407] Connection header received
0926 13:45:28.488130 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:28.488148 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:28.488492 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:28.488753 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:28.489420 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:28.490824 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:28.491712 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:28.492408 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:28.492572 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:28.494819 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:28.494824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:28.494837 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:28.494868 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:28.496524 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:28.497026 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:28.497031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:28.497037 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:28.497160 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:28.497465 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:28.497468 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:28.497470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:28.497924 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:28.498634 (+ 710us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:28.498794 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 13:45:28.499064 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":116,"threads_started":1}
I20260926 13:45:28.501684 21233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:45:28.506381 21233 catalog_manager.cc:1347] Generated new cluster ID: b0c66b8a28854867994240734039749d
I20260926 13:45:28.506438 21233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:45:28.519598 21233 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:45:28.520443 21233 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:45:28.526109 21233 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd: Generated new TSK 0
I20260926 13:45:28.526768 21233 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.170.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
I20260926 13:45:28.558565 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:0
--local_ip_for_outbound_sockets=127.12.170.65
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:40817
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:39721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:28.661083 21245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:28.661316 21245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:28.661371 21245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:28.665237 21245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:28.665288 21245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:28.665364 21245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:45:28.670281 21245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:39721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:40817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:28.671190 21245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:28.672148 21245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:28.680930 21253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:28.680867 21250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:28.680867 21251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:28.680975 21245 server_base.cc:1034] running on GCE node
I20260926 13:45:28.681421 21245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:28.681946 21245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:28.683092 21245 hybrid_clock.cc:648] HybridClock initialized: now 1790430328683062 us; error 42 us; skew 500 ppm
Sep 26 13:45:28 dist-test-slave-hc1k krb5kdc[21138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430328, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:28.685513 21245 init.cc:377] Logged in from keytab as kudu/127.12.170.65@KRBTEST.COM (short username kudu)
I20260926 13:45:28.686264 21245 webserver.cc:460] Webserver started at http://127.12.170.65:44799/ using document root <none> and password file <none>
I20260926 13:45:28.686740 21245 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:28.686797 21245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:28.686975 21245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:28.688829 21245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/data/instance:
uuid: "6b96aea08d574d98991a1aafd6741802"
format_stamp: "Formatted at 2026-09-26 13:45:28 on dist-test-slave-hc1k"
server_key: "eafbff5420a430773cae0bc768de9e91"
server_key_iv: "24047af43afbbab9d8075fbe5caca3b4"
server_key_version: "encryptionkey@0"
I20260926 13:45:28.689318 21245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance:
uuid: "6b96aea08d574d98991a1aafd6741802"
format_stamp: "Formatted at 2026-09-26 13:45:28 on dist-test-slave-hc1k"
server_key: "eafbff5420a430773cae0bc768de9e91"
server_key_iv: "24047af43afbbab9d8075fbe5caca3b4"
server_key_version: "encryptionkey@0"
I20260926 13:45:28.692574 21245 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:45:28.694649 21260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:28.695744 21245 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:45:28.695838 21245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "6b96aea08d574d98991a1aafd6741802"
format_stamp: "Formatted at 2026-09-26 13:45:28 on dist-test-slave-hc1k"
server_key: "eafbff5420a430773cae0bc768de9e91"
server_key_iv: "24047af43afbbab9d8075fbe5caca3b4"
server_key_version: "encryptionkey@0"
I20260926 13:45:28.695926 21245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:28.709578 21245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:28.711908 21245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:28.712049 21245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:28.712527 21245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:28.713217 21245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:28.713259 21245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:28.713326 21245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:28.713356 21245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:28.721194 21245 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:43299
I20260926 13:45:28.721233 21373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:43299 every 8 connection(s)
I20260926 13:45:28.721966 21245 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:45:28.723562 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 21245
I20260926 13:45:28.723699 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance
I20260926 13:45:28.723997 12969 external_mini_cluster.cc:1428] Setting key c0d1d57e0a8e1a5d168421ed42f4b4bb
Sep 26 13:45:28 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430328, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.170.66@KRBTEST.COM; defaulting to no policy
I20260926 13:45:28.735340 21222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:28.723498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:55029 (local address 127.12.170.126:40817)
0926 13:45:28.723610 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:28.723615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:28.724552 (+ 937us) server_negotiation.cc:407] Connection header received
0926 13:45:28.725934 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:28.725938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:28.725991 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:28.726062 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:28.727234 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:28.727822 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:28.729321 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:28.729539 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:28.729699 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:28.732304 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:28.732306 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:28.732309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:28.732332 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:28.733614 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:28.734091 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:28.734094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:28.734095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:28.734166 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:28.734413 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:28.734415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:28.734417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:28.734564 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:28.734631 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:28.735098 (+ 467us) server_negotiation.cc:299] Negotiation successful
0926 13:45:28.735194 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.66@KRBTEST.COM" created.
I20260926 13:45:28.736111 21376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:28.723937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.65:55029)
0926 13:45:28.724391 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:28.724442 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:28.725746 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:28.726212 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:28.726218 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:28.726524 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:28.727091 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:28.727107 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:28.727934 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:28.727937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:28.729205 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:28.729217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:28.729751 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:28.729759 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:28.729964 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:28.730511 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:28.730530 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:28.732129 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:28.733718 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:28.733722 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:28.733733 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:28.734006 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:28.734235 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:28.734237 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:28.734238 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:28.734352 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:28.734631 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:28.734635 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:28.734905 (+ 270us) client_negotiation.cc:769] Sending connection context
0926 13:45:28.735046 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 13:45:28.735326 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":136,"threads_started":1}
I20260926 13:45:28.737191 21374 heartbeater.cc:347] Connected to a master server at 127.12.170.126:40817
I20260926 13:45:28.737396 21374 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:28.737811 21374 heartbeater.cc:511] Master 127.12.170.126:40817 requested a full tablet report, sending...
I20260926 13:45:28.739156 21183 ts_manager.cc:194] Registered new tserver with Master: 6b96aea08d574d98991a1aafd6741802 (127.12.170.65:43299)
I20260926 13:45:28.740222 21183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.65@KRBTEST.COM'} at 127.12.170.65:55029
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
I20260926 13:45:28.748117 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:0
--local_ip_for_outbound_sockets=127.12.170.66
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:40817
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:39721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:28.851895 21379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:28.852100 21379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:28.852138 21379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:28.855995 21379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:28.856042 21379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:28.856094 21379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:45:28.860965 21379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:39721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:40817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:28.861773 21379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:28.862584 21379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:28.870936 21385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:28.870944 21384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:28.870944 21387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:28.871129 21379 server_base.cc:1034] running on GCE node
I20260926 13:45:28.871483 21379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:28.872108 21379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:28.873279 21379 hybrid_clock.cc:648] HybridClock initialized: now 1790430328873259 us; error 33 us; skew 500 ppm
Sep 26 13:45:28 dist-test-slave-hc1k krb5kdc[21138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430328, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:28.875820 21379 init.cc:377] Logged in from keytab as kudu/127.12.170.66@KRBTEST.COM (short username kudu)
I20260926 13:45:28.876598 21379 webserver.cc:460] Webserver started at http://127.12.170.66:40881/ using document root <none> and password file <none>
I20260926 13:45:28.877095 21379 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:28.877180 21379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:28.877394 21379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:28.879314 21379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/data/instance:
uuid: "053eccb0c49640629923843714eb5e55"
format_stamp: "Formatted at 2026-09-26 13:45:28 on dist-test-slave-hc1k"
server_key: "b597f7a1ceffcda2ee60f408eb523c0d"
server_key_iv: "56dab2f24fcf2947281b48c7af7b93f6"
server_key_version: "encryptionkey@0"
I20260926 13:45:28.879873 21379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance:
uuid: "053eccb0c49640629923843714eb5e55"
format_stamp: "Formatted at 2026-09-26 13:45:28 on dist-test-slave-hc1k"
server_key: "b597f7a1ceffcda2ee60f408eb523c0d"
server_key_iv: "56dab2f24fcf2947281b48c7af7b93f6"
server_key_version: "encryptionkey@0"
I20260926 13:45:28.883282 21379 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 13:45:28.885540 21394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:28.886705 21379 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:28.886811 21379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "053eccb0c49640629923843714eb5e55"
format_stamp: "Formatted at 2026-09-26 13:45:28 on dist-test-slave-hc1k"
server_key: "b597f7a1ceffcda2ee60f408eb523c0d"
server_key_iv: "56dab2f24fcf2947281b48c7af7b93f6"
server_key_version: "encryptionkey@0"
I20260926 13:45:28.886893 21379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:28.905895 21379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:28.908828 21379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:28.908985 21379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:28.909509 21379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:28.910262 21379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:28.910308 21379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:28.910357 21379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:28.910386 21379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:28.918390 21379 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:33191
I20260926 13:45:28.918449 21507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:33191 every 8 connection(s)
I20260926 13:45:28.919224 21379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:45:28.922859 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 21379
I20260926 13:45:28.922979 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance
I20260926 13:45:28.923297 12969 external_mini_cluster.cc:1428] Setting key 9fbddd8be4d5e788c44ade22c1781627
Sep 26 13:45:28 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430328, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.170.67@KRBTEST.COM; defaulting to no policy
I20260926 13:45:28.933135 21222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:28.920979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:50333 (local address 127.12.170.126:40817)
0926 13:45:28.921099 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:28.921102 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:28.921892 (+ 790us) server_negotiation.cc:407] Connection header received
0926 13:45:28.923284 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:28.923287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:28.923337 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:28.923434 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:28.924700 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:28.925278 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:28.926707 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:28.926927 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:28.927015 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:28.929491 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:28.929495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:28.929498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:28.929529 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:28.931083 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:28.931579 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:28.931582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:28.931584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:28.931688 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:28.932082 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:28.932085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:28.932087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:28.932317 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:28.932419 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:28.932924 (+ 505us) server_negotiation.cc:299] Negotiation successful
0926 13:45:28.933018 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:45:28.933909 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:28.921330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.66:50333)
0926 13:45:28.921752 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:28.921800 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:28.923043 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:28.923632 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:28.923641 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:28.924072 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:28.924556 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:28.924575 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:28.925406 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:28.925410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:28.926597 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:28.926606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:28.927056 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:28.927059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:28.927212 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:28.927756 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:28.927774 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:28.929321 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:28.931225 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:28.931230 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:28.931242 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:28.931482 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:28.931840 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:28.931843 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:28.931845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:28.931981 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:28.932440 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:28.932446 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:28.932681 (+ 235us) client_negotiation.cc:769] Sending connection context
0926 13:45:28.932861 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 13:45:28.933101 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":126,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.67@KRBTEST.COM" created.
I20260926 13:45:28.934988 21508 heartbeater.cc:347] Connected to a master server at 127.12.170.126:40817
I20260926 13:45:28.935204 21508 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:28.935684 21508 heartbeater.cc:511] Master 127.12.170.126:40817 requested a full tablet report, sending...
I20260926 13:45:28.936630 21183 ts_manager.cc:194] Registered new tserver with Master: 053eccb0c49640629923843714eb5e55 (127.12.170.66:33191)
I20260926 13:45:28.937166 21183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.66@KRBTEST.COM'} at 127.12.170.66:50333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
I20260926 13:45:28.948369 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:0
--local_ip_for_outbound_sockets=127.12.170.67
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:40817
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:39721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:29.052284 21513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:29.052542 21513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:29.052589 21513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:29.056449 21513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:29.056504 21513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:29.056558 21513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:45:29.061638 21513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:39721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:40817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:29.062619 21513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:29.063581 21513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:29.072247 21518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:29.072296 21519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:29.072345 21521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:29.072484 21513 server_base.cc:1034] running on GCE node
I20260926 13:45:29.072870 21513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:29.073406 21513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:29.074569 21513 hybrid_clock.cc:648] HybridClock initialized: now 1790430329074550 us; error 28 us; skew 500 ppm
Sep 26 13:45:29 dist-test-slave-hc1k krb5kdc[21138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430329, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:29.077065 21513 init.cc:377] Logged in from keytab as kudu/127.12.170.67@KRBTEST.COM (short username kudu)
I20260926 13:45:29.077800 21513 webserver.cc:460] Webserver started at http://127.12.170.67:40451/ using document root <none> and password file <none>
I20260926 13:45:29.078276 21513 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:29.078328 21513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:29.078480 21513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:29.080360 21513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/data/instance:
uuid: "3ff1b34c32044c5b9b4f0ff77a711340"
format_stamp: "Formatted at 2026-09-26 13:45:29 on dist-test-slave-hc1k"
server_key: "08df3a201fd2477f1c67b53c1986b501"
server_key_iv: "e0b68650a6f1b1583e13243f27fdddec"
server_key_version: "encryptionkey@0"
I20260926 13:45:29.080821 21513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance:
uuid: "3ff1b34c32044c5b9b4f0ff77a711340"
format_stamp: "Formatted at 2026-09-26 13:45:29 on dist-test-slave-hc1k"
server_key: "08df3a201fd2477f1c67b53c1986b501"
server_key_iv: "e0b68650a6f1b1583e13243f27fdddec"
server_key_version: "encryptionkey@0"
I20260926 13:45:29.084106 21513 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:45:29.086241 21528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:29.087320 21513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:29.087407 21513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "3ff1b34c32044c5b9b4f0ff77a711340"
format_stamp: "Formatted at 2026-09-26 13:45:29 on dist-test-slave-hc1k"
server_key: "08df3a201fd2477f1c67b53c1986b501"
server_key_iv: "e0b68650a6f1b1583e13243f27fdddec"
server_key_version: "encryptionkey@0"
I20260926 13:45:29.087481 21513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:29.100121 21513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:29.102530 21513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:29.102658 21513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:29.103109 21513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:29.103819 21513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:29.103863 21513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:29.103920 21513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:29.103941 21513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:29.111500 21513 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:45939
I20260926 13:45:29.111531 21641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:45939 every 8 connection(s)
I20260926 13:45:29.112284 21513 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:45:29.113364 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 21513
I20260926 13:45:29.113463 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance
I20260926 13:45:29.113695 12969 external_mini_cluster.cc:1428] Setting key 22f5100a35f86d55364d9f1633ac9f2b
Sep 26 13:45:29 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430329, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:29.123859 21222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:29.113930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:39535 (local address 127.12.170.126:40817)
0926 13:45:29.114067 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:29.114071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:29.114915 (+ 844us) server_negotiation.cc:407] Connection header received
0926 13:45:29.116192 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:29.116194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:29.116238 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:29.116299 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:29.117282 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.117792 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.118793 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.118985 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.119049 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:29.121101 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:29.121104 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:29.121106 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:29.121128 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:29.122402 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:29.122764 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:29.122766 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:29.122767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:29.122830 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:29.123058 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:29.123060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:29.123061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:29.123204 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:29.123270 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:29.123631 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 13:45:29.123749 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 13:45:29.124543 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:29.114333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.67:39535)
0926 13:45:29.114769 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:29.114816 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:29.116017 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:29.116425 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:29.116431 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:29.116747 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:29.117178 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:29.117191 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.117873 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.117875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:29.118708 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:29.118717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.119079 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.119081 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:29.119207 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:29.119689 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:29.119708 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:29.120966 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:29.122479 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:29.122483 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:29.122492 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:29.122700 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:29.122887 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:29.122889 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:29.122890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:29.122995 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:29.123274 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:29.123279 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:29.123466 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 13:45:29.123600 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 13:45:29.123844 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":108,"threads_started":1}
I20260926 13:45:29.125480 21642 heartbeater.cc:347] Connected to a master server at 127.12.170.126:40817
I20260926 13:45:29.125658 21642 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:29.126103 21642 heartbeater.cc:511] Master 127.12.170.126:40817 requested a full tablet report, sending...
I20260926 13:45:29.126976 21183 ts_manager.cc:194] Registered new tserver with Master: 3ff1b34c32044c5b9b4f0ff77a711340 (127.12.170.67:45939)
I20260926 13:45:29.127542 21183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.67@KRBTEST.COM'} at 127.12.170.67:39535
I20260926 13:45:29.138737 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.170.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.170.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.170.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.65.keytab.
Entry for principal oryx/127.12.170.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.65.keytab.
WARNING: no policy specified for oryx/127.12.170.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.170.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.170.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.66.keytab.
Entry for principal oryx/127.12.170.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.66.keytab.
WARNING: no policy specified for oryx/127.12.170.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.170.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.170.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.67.keytab.
Entry for principal oryx/127.12.170.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.67.keytab.
I20260926 13:45:29.199431 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 21245
I20260926 13:45:29.204794 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 21379
I20260926 13:45:29.208678 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 21513
I20260926 13:45:29.212746 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 21151
I20260926 13:45:29.217320 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:40817
--webserver_interface=127.12.170.126
--webserver_port=42973
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:39721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:40817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:29.321341 21651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:29.321573 21651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:29.321626 21651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:29.325611 21651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:45:29.325663 21651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:29.325687 21651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:29.325716 21651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:45:29.325742 21651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:45:29.330703 21651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:39721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:40817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:40817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=42973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:29.331598 21651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:29.332512 21651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:29.340034 21657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:29.340044 21659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:29.340034 21656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:29.340161 21651 server_base.cc:1034] running on GCE node
I20260926 13:45:29.340762 21651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:29.341749 21651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:29.342909 21651 hybrid_clock.cc:648] HybridClock initialized: now 1790430329342892 us; error 31 us; skew 500 ppm
Sep 26 13:45:29 dist-test-slave-hc1k krb5kdc[21138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430329, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:29.345422 21651 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:45:29.346172 21651 webserver.cc:460] Webserver started at http://127.12.170.126:42973/ using document root <none> and password file <none>
I20260926 13:45:29.346647 21651 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:29.346709 21651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:29.350168 21651 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:29.352159 21666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:29.353392 21651 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:29.353488 21651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "815f6b54c3ef4096823eda6e73eb0efd"
format_stamp: "Formatted at 2026-09-26 13:45:28 on dist-test-slave-hc1k"
server_key: "b0ee49eb003808b91e0840776c65c656"
server_key_iv: "886a34bdbcfe31396012dfc1f9c02657"
server_key_version: "encryptionkey@0"
I20260926 13:45:29.353960 21651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:29.371269 21651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:29.374116 21651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:29.374269 21651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:29.380909 21651 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:40817
I20260926 13:45:29.380928 21718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:40817 every 8 connection(s)
I20260926 13:45:29.381907 21651 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:45:29.382272 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 21651
I20260926 13:45:29.382702 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:43299
--local_ip_for_outbound_sockets=127.12.170.65
--tserver_master_addrs=127.12.170.126:40817
--webserver_port=44799
--webserver_interface=127.12.170.65
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:39721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.170.65
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.65.keytab with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:45:29.386868 21719 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:45:29.389007 21719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd: Bootstrap starting.
I20260926 13:45:29.409998 21719 log.cc:824] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:29.417732 21719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:45:29.418098 21719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd: Bootstrap complete.
I20260926 13:45:29.423305 21719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815f6b54c3ef4096823eda6e73eb0efd" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40817 } }
I20260926 13:45:29.423854 21719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 815f6b54c3ef4096823eda6e73eb0efd, State: Initialized, Role: FOLLOWER
I20260926 13:45:29.424378 21719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815f6b54c3ef4096823eda6e73eb0efd" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40817 } }
I20260926 13:45:29.424517 21719 raft_consensus.cc:388] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:45:29.424582 21719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:45:29.424669 21719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [term 1 FOLLOWER]: Advancing to term 2
I20260926 13:45:29.426972 21719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815f6b54c3ef4096823eda6e73eb0efd" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40817 } }
I20260926 13:45:29.427305 21719 leader_election.cc:302] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 815f6b54c3ef4096823eda6e73eb0efd; no voters:
I20260926 13:45:29.427614 21719 leader_election.cc:288] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 13:45:29.427778 21723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [term 2 FOLLOWER]: Leader election won for term 2
I20260926 13:45:29.427994 21723 raft_consensus.cc:686] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [term 2 LEADER]: Becoming Leader. State: Replica: 815f6b54c3ef4096823eda6e73eb0efd, State: Running, Role: LEADER
I20260926 13:45:29.428293 21723 consensus_queue.cc:237] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815f6b54c3ef4096823eda6e73eb0efd" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40817 } }
I20260926 13:45:29.428808 21719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:45:29.429162 21725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 815f6b54c3ef4096823eda6e73eb0efd. Latest consensus state: current_term: 2 leader_uuid: "815f6b54c3ef4096823eda6e73eb0efd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815f6b54c3ef4096823eda6e73eb0efd" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40817 } } }
I20260926 13:45:29.429296 21725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:29.429508 21724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "815f6b54c3ef4096823eda6e73eb0efd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "815f6b54c3ef4096823eda6e73eb0efd" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40817 } } }
I20260926 13:45:29.429582 21724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:29.431016 21727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:45:29.434257 21727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:45:29.434864 21727 catalog_manager.cc:1259] Loaded cluster ID: b0c66b8a28854867994240734039749d
I20260926 13:45:29.434919 21727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:45:29.437114 21727 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:45:29.437723 21727 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 815f6b54c3ef4096823eda6e73eb0efd: Loaded TSK: 0
I20260926 13:45:29.438882 21727 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 13:45:29.503877 21721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:29.504089 21721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:29.504127 21721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:29.508001 21721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:29.508061 21721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:29.508133 21721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:45:29.513108 21721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:39721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.65.keytab
--principal=oryx/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:43299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=44799
--tserver_master_addrs=127.12.170.126:40817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:29.513962 21721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:29.514839 21721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:29.523643 21745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:29.523625 21746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:29.523793 21748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:29.524597 21721 server_base.cc:1034] running on GCE node
I20260926 13:45:29.524956 21721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:29.525498 21721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:29.526647 21721 hybrid_clock.cc:648] HybridClock initialized: now 1790430329526623 us; error 45 us; skew 500 ppm
Sep 26 13:45:29 dist-test-slave-hc1k krb5kdc[21138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430329, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:29.529351 21721 init.cc:377] Logged in from keytab as oryx/127.12.170.65@KRBTEST.COM (short username oryx)
I20260926 13:45:29.530236 21721 webserver.cc:460] Webserver started at http://127.12.170.65:44799/ using document root <none> and password file <none>
I20260926 13:45:29.530782 21721 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:29.530861 21721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:29.534597 21721 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:29.536592 21755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:29.537750 21721 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:29.537838 21721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "6b96aea08d574d98991a1aafd6741802"
format_stamp: "Formatted at 2026-09-26 13:45:28 on dist-test-slave-hc1k"
server_key: "eafbff5420a430773cae0bc768de9e91"
server_key_iv: "24047af43afbbab9d8075fbe5caca3b4"
server_key_version: "encryptionkey@0"
I20260926 13:45:29.538326 21721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:29.556547 21721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:29.559386 21721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:29.559541 21721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:29.560107 21721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:29.560870 21721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:29.560913 21721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:29.560950 21721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:29.560979 21721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:29.569547 21721 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:43299
I20260926 13:45:29.569712 21868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:43299 every 8 connection(s)
I20260926 13:45:29.570487 21721 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:45:29.579118 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 21721
I20260926 13:45:29.579495 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:33191
--local_ip_for_outbound_sockets=127.12.170.66
--tserver_master_addrs=127.12.170.126:40817
--webserver_port=40881
--webserver_interface=127.12.170.66
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:39721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.170.66
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.66.keytab with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:45:29 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:29 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 13:45:29.582991 21871 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:29.572394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:36437 (local address 127.12.170.126:40817)
0926 13:45:29.572884 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:29.572905 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:29.573107 (+ 202us) server_negotiation.cc:407] Connection header received
0926 13:45:29.574550 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:29.574561 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:29.574860 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:29.575206 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:29.576358 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.577300 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.578441 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.578912 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.579072 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:29.582323 (+ 3251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.65:36437: BlockingRecv error: recv got EOF from 127.12.170.65:36437 (error 108)
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":105,"threads_started":1}
I20260926 13:45:29.583003 21872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:29.572514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.65:36437)
0926 13:45:29.572954 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:29.572999 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:29.574146 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:29.575308 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:29.575317 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:29.575736 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:29.576172 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:29.576197 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.577442 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.577445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:29.578307 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:29.578318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.579073 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.579078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:29.579271 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:29.579876 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:29.579893 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:29.582101 (+ 2208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":120,"threads_started":1}
W20260926 13:45:29.583643 21869 heartbeater.cc:644] Failed to heartbeat to 127.12.170.126:40817 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.170.126:40817: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Sep 26 13:45:29 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:29 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:29.588723 21872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:29.584064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.65:45529)
0926 13:45:29.584237 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:29.584252 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:29.584341 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:29.584580 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:29.584584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:29.584724 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:29.584811 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:29.584816 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.585517 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.585522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:29.586050 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:29.586057 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.586479 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.586483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:29.586573 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:29.586993 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:29.587005 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:29.588599 (+ 1594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94}
W20260926 13:45:29.588852 21871 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:29.584154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:45529 (local address 127.12.170.126:40817)
0926 13:45:29.584284 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:29.584294 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:29.584337 (+ 43us) server_negotiation.cc:407] Connection header received
0926 13:45:29.584429 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:29.584432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:29.584477 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:29.584545 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:29.584917 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.585405 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.586160 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.586381 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.586475 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:29.588746 (+ 2271us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.65:45529: BlockingRecv error: recv got EOF from 127.12.170.65:45529 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 13:45:29 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:29 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:29.594225 21872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:29.589331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.65:53373)
0926 13:45:29.589429 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:29.589443 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:29.589511 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:29.589756 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:29.589758 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:29.589871 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:29.589939 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:29.589944 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.590891 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.590893 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:29.591477 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:29.591485 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.592032 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.592035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:29.592102 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:29.592519 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:29.592530 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:29.594117 (+ 1587us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 13:45:29.594429 21869 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:45:29.594558 21871 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:29.589404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:53373 (local address 127.12.170.126:40817)
0926 13:45:29.589504 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:29.589508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:29.589521 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:45:29.589583 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:29.589585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:29.589624 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:29.589703 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:29.590027 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.590790 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.591566 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.591914 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.592007 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:29.594226 (+ 2219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.65:53373: BlockingRecv error: recv got EOF from 127.12.170.65:53373 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 13:45:29.683590 21873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:29.683866 21873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:29.683917 21873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:29.687790 21873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:29.687839 21873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:29.687903 21873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:45:29.692878 21873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:39721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.66.keytab
--principal=oryx/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:33191
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=40881
--tserver_master_addrs=127.12.170.126:40817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:29.693784 21873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:29.694725 21873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:29.703334 21881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:29.703375 21878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:29.703370 21879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:29.703439 21873 server_base.cc:1034] running on GCE node
I20260926 13:45:29.704046 21873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:29.704578 21873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:29.705725 21873 hybrid_clock.cc:648] HybridClock initialized: now 1790430329705709 us; error 33 us; skew 500 ppm
Sep 26 13:45:29 dist-test-slave-hc1k krb5kdc[21138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430329, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:29.708184 21873 init.cc:377] Logged in from keytab as oryx/127.12.170.66@KRBTEST.COM (short username oryx)
I20260926 13:45:29.708948 21873 webserver.cc:460] Webserver started at http://127.12.170.66:40881/ using document root <none> and password file <none>
I20260926 13:45:29.709424 21873 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:29.709496 21873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:29.712934 21873 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:45:29.714728 21888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:29.715860 21873 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:29.715953 21873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "053eccb0c49640629923843714eb5e55"
format_stamp: "Formatted at 2026-09-26 13:45:28 on dist-test-slave-hc1k"
server_key: "b597f7a1ceffcda2ee60f408eb523c0d"
server_key_iv: "56dab2f24fcf2947281b48c7af7b93f6"
server_key_version: "encryptionkey@0"
I20260926 13:45:29.716413 21873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:29.740552 21873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:29.743140 21873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:29.743290 21873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:29.743835 21873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:29.744575 21873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:29.744618 21873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:29.744669 21873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:29.744699 21873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:29.752746 21873 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:33191
I20260926 13:45:29.752803 22001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:33191 every 8 connection(s)
I20260926 13:45:29.753660 21873 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:45:29.755071 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 21873
I20260926 13:45:29.755523 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:45939
--local_ip_for_outbound_sockets=127.12.170.67
--tserver_master_addrs=127.12.170.126:40817
--webserver_port=40451
--webserver_interface=127.12.170.67
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:39721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.170.67
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.67.keytab with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:45:29 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:29 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 13:45:29.764544 21871 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:29.755343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:60957 (local address 127.12.170.126:40817)
0926 13:45:29.755490 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:29.755495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:29.756404 (+ 909us) server_negotiation.cc:407] Connection header received
0926 13:45:29.757914 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:29.757918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:29.757981 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:29.758069 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:29.759135 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.759679 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.760767 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.760982 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.761071 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:29.764411 (+ 3340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.66:60957: BlockingRecv error: recv got EOF from 127.12.170.66:60957 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 13:45:29.764943 22004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:29.755764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.66:60957)
0926 13:45:29.756231 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:29.756288 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:29.757714 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:29.758217 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:29.758224 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:29.758544 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:29.758992 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:29.759006 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.759794 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.759798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:29.760647 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:29.760655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.761108 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.761112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:29.761270 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:29.761847 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:29.761861 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:29.764221 (+ 2360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":108,"threads_started":1}
W20260926 13:45:29.765458 22002 heartbeater.cc:644] Failed to heartbeat to 127.12.170.126:40817 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.170.126:40817: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Sep 26 13:45:29 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:29 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:29.770309 22004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:29.765859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.66:52587)
0926 13:45:29.765940 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:29.765953 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:29.766018 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:29.766228 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:29.766231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:29.766374 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:29.766453 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:29.766458 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.767116 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.767119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:29.767645 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:29.767675 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.768066 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.768069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:29.768151 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:29.768590 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:29.768600 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:29.770189 (+ 1589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 13:45:29.770418 21871 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:29.765955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:52587 (local address 127.12.170.126:40817)
0926 13:45:29.766063 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:29.766067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:29.766078 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:45:29.766110 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:29.766112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:29.766145 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:29.766205 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:29.766551 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.767026 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.767772 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.767977 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.768059 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:29.770317 (+ 2258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.66:52587: BlockingRecv error: recv got EOF from 127.12.170.66:52587 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 26 13:45:29 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:29 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:29.775352 22004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:29.770936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.66:42371)
0926 13:45:29.771134 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:29.771147 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:29.771221 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:29.771415 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:29.771418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:29.771551 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:29.771611 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:29.771615 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.772309 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.772312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:29.772822 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:29.772829 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.773202 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.773205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:29.773273 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:29.773678 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:29.773687 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:29.775237 (+ 1550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144}
W20260926 13:45:29.775524 21871 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:29.771035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:42371 (local address 127.12.170.126:40817)
0926 13:45:29.771134 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:29.771138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:29.771196 (+ 58us) server_negotiation.cc:407] Connection header received
0926 13:45:29.771298 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:29.771301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:29.771335 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:29.771392 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:29.771733 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.772215 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.772918 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.773116 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.773195 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:29.775352 (+ 2157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.66:42371: BlockingRecv error: recv got EOF from 127.12.170.66:42371 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 13:45:29.775732 22002 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:45:29.862989 22005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:29.863207 22005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:29.863260 22005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:29.867197 22005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:29.867249 22005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:29.867321 22005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:45:29.872268 22005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:39721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/krb5kdc/oryx_127.12.170.67.keytab
--principal=oryx/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:45939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=40451
--tserver_master_addrs=127.12.170.126:40817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:29.873137 22005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:29.874027 22005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:29.882900 22010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:29.882953 22005 server_base.cc:1034] running on GCE node
W20260926 13:45:29.882858 22013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:29.882858 22011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:29.883445 22005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:29.884079 22005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:29.885236 22005 hybrid_clock.cc:648] HybridClock initialized: now 1790430329885219 us; error 27 us; skew 500 ppm
Sep 26 13:45:29 dist-test-slave-hc1k krb5kdc[21138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430329, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:29.887842 22005 init.cc:377] Logged in from keytab as oryx/127.12.170.67@KRBTEST.COM (short username oryx)
I20260926 13:45:29.888634 22005 webserver.cc:460] Webserver started at http://127.12.170.67:40451/ using document root <none> and password file <none>
I20260926 13:45:29.889124 22005 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:29.889189 22005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:29.892678 22005 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 13:45:29.894562 22020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:29.895709 22005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:29.895804 22005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "3ff1b34c32044c5b9b4f0ff77a711340"
format_stamp: "Formatted at 2026-09-26 13:45:29 on dist-test-slave-hc1k"
server_key: "08df3a201fd2477f1c67b53c1986b501"
server_key_iv: "e0b68650a6f1b1583e13243f27fdddec"
server_key_version: "encryptionkey@0"
I20260926 13:45:29.896279 22005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:29.923826 22005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:29.926393 22005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:29.926544 22005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:29.927091 22005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:29.927882 22005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:29.927927 22005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:29.927971 22005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:29.927999 22005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:29.936177 22005 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:45939
I20260926 13:45:29.936209 22133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:45939 every 8 connection(s)
I20260926 13:45:29.937069 22005 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:45:29.941511 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 22005
Sep 26 13:45:29 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:29 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 13:45:29.948318 21871 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:29.938652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:58255 (local address 127.12.170.126:40817)
0926 13:45:29.938801 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:29.938805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:29.939539 (+ 734us) server_negotiation.cc:407] Connection header received
0926 13:45:29.940774 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:29.940777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:29.940821 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:29.940891 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:29.942383 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.943084 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.944535 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.944883 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.944986 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:29.948202 (+ 3216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.67:58255: BlockingRecv error: recv got EOF from 127.12.170.67:58255 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 13:45:29.948935 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:29.939003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.67:58255)
0926 13:45:29.939389 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:29.939430 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:29.940600 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:29.941079 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:29.941086 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:29.941588 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:29.942212 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:29.942234 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.943207 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.943210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:29.944428 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:29.944437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.945008 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.945010 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:29.945144 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:29.945645 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:29.945665 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:29.948103 (+ 2438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":104,"threads_started":1}
W20260926 13:45:29.949342 22134 heartbeater.cc:644] Failed to heartbeat to 127.12.170.126:40817 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.170.126:40817: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
I20260926 13:45:29.950810 22138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:29.942530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49766 (local address 127.12.170.126:40817)
0926 13:45:29.942732 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:29.942736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:29.942749 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:45:29.942783 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:29.942786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:29.942825 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:29.942911 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:29.943592 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.944389 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.945391 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.945652 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.945755 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:29.947046 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:29.947051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:29.947063 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:29.947086 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:29.948680 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:29.949133 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:29.949140 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:29.949146 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:29.949270 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:29.949542 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:29.949546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:29.949548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:29.949985 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:29.950157 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:29.950416 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 13:45:29.950628 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":62,"threads_started":1}
Sep 26 13:45:29 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:29 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:29.954084 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:29.949681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.67:46303)
0926 13:45:29.949840 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:29.949851 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:29.949905 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:29.950077 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:29.950080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:29.950220 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:29.950320 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:29.950326 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.951146 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.951149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:29.951716 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:29.951726 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.952112 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.952116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:29.952199 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:29.952561 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:29.952574 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:29.954006 (+ 1432us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103}
W20260926 13:45:29.954187 21871 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:29.949765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:46303 (local address 127.12.170.126:40817)
0926 13:45:29.949868 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:29.949871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:29.949884 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:45:29.949962 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:29.949964 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:29.949999 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:29.950060 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:29.950411 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.951050 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.951822 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.952026 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.952102 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:29.954110 (+ 2008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.67:46303: BlockingRecv error: recv got EOF from 127.12.170.67:46303 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 13:45:29 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:29 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:29.958472 22136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:29.954491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.67:53891)
0926 13:45:29.954576 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:29.954588 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:29.954642 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:29.954795 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:29.954797 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:29.954904 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:29.954964 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:29.954968 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.955555 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.955558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:29.956107 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:29.956116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.956497 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.956500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:29.956581 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:29.956903 (+ 322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:29.956914 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:29.958395 (+ 1481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 13:45:29.958590 21871 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:29.954551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:53891 (local address 127.12.170.126:40817)
0926 13:45:29.954656 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:29.954659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:29.954670 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:45:29.954695 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:29.954697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:29.954730 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:29.954787 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:29.955034 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.955475 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.956196 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:29.956392 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:29.956455 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:29.958502 (+ 2047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.67:53891: BlockingRecv error: recv got EOF from 127.12.170.67:53891 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 13:45:29.958634 22134 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 13:45:30 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:30 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:30.600541 22139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:30.595016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.65:40867)
0926 13:45:30.595277 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:30.595291 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:30.595380 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:30.595589 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:30.595591 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:30.595785 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:30.595898 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:30.595904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:30.596646 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:30.596650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:30.597259 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:30.597270 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:30.597723 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:30.597725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:30.597804 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:30.598268 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:30.598280 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:30.600423 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":86,"threads_started":1}
W20260926 13:45:30.600651 22140 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:30.595155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:40867 (local address 127.12.170.126:40817)
0926 13:45:30.595407 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:30.595410 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:30.595422 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:45:30.595455 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:30.595457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:30.595495 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:30.595566 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:30.596001 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:30.596521 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:30.597386 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:30.597613 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:30.597700 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:30.600542 (+ 2842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.65:40867: BlockingRecv error: recv got EOF from 127.12.170.65:40867 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":105,"threads_started":1}
Sep 26 13:45:30 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:30 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:30.781406 22141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:30.776220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.66:60411)
0926 13:45:30.776492 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:30.776507 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:30.776617 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:30.776846 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:30.776849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:30.777032 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:30.777119 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:30.777125 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:30.777852 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:30.777855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:30.778432 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:30.778440 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:30.778854 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:30.778858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:30.778938 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:30.779400 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:30.779414 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:30.781292 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":90,"threads_started":1}
W20260926 13:45:30.781518 22140 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:30.776310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:60411 (local address 127.12.170.126:40817)
0926 13:45:30.776441 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:30.776447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:30.776569 (+ 122us) server_negotiation.cc:407] Connection header received
0926 13:45:30.776698 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:30.776702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:30.776750 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:30.776814 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:30.777221 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:30.777746 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:30.778538 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:30.778757 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:30.778842 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:30.781411 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.66:60411: BlockingRecv error: recv got EOF from 127.12.170.66:60411 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 13:45:30 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:30 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:30.964084 22142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:30.959155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.67:33365)
0926 13:45:30.959383 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:30.959395 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:30.959484 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:30.959706 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:30.959708 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:30.959874 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:30.959990 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:30.959996 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:30.960675 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:30.960678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:30.961252 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:30.961260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:30.961665 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:30.961667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:30.961736 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:30.962165 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:30.962176 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:30.963977 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":79,"threads_started":1}
W20260926 13:45:30.964236 22140 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:30.959171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:33365 (local address 127.12.170.126:40817)
0926 13:45:30.959289 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:30.959292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:30.959452 (+ 160us) server_negotiation.cc:407] Connection header received
0926 13:45:30.959553 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:30.959555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:30.959602 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:30.959692 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:30.960084 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:30.960591 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:30.961355 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:30.961582 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:30.961657 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:30.964112 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.67:33365: BlockingRecv error: recv got EOF from 127.12.170.67:33365 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 13:45:31 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:31 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:31.606506 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:31.601275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.65:40027)
0926 13:45:31.601548 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:31.601562 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:31.601650 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:31.601884 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:31.601886 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:31.602065 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:31.602166 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:31.602181 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:31.602939 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:31.602941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:31.603512 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:31.603521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:31.603974 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:31.603976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:31.604043 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:31.604538 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:31.604550 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:31.606395 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":83,"threads_started":1}
W20260926 13:45:31.606632 22144 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:31.601343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:40027 (local address 127.12.170.126:40817)
0926 13:45:31.601607 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:31.601610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:31.601623 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:45:31.601753 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:31.601755 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:31.601797 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:31.601872 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:31.602295 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:31.602840 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:31.603620 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:31.603885 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:31.603960 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:31.606509 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.65:40027: BlockingRecv error: recv got EOF from 127.12.170.65:40027 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":108,"threads_started":1}
Sep 26 13:45:31 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:31 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:31.788378 22145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:31.782100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.66:55939)
0926 13:45:31.782335 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:31.782348 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:31.782456 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:31.782705 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:31.782707 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:31.782910 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:31.783023 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:31.783030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:31.784056 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:31.784059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:31.784936 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:31.784945 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:31.785492 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:31.785495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:31.785576 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:31.786083 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:31.786097 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:31.788192 (+ 2095us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":73,"threads_started":1}
W20260926 13:45:31.788826 22144 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:31.782117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:55939 (local address 127.12.170.126:40817)
0926 13:45:31.782234 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:31.782238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:31.782426 (+ 188us) server_negotiation.cc:407] Connection header received
0926 13:45:31.782537 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:31.782540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:31.782598 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:31.782677 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:31.783129 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:31.783966 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:31.785047 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:31.785394 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:31.785498 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:31.788324 (+ 2826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.66:55939: BlockingRecv error: recv got EOF from 127.12.170.66:55939 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 26 13:45:31 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:31 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:31.969900 22146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:31.964786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.67:44835)
0926 13:45:31.965029 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:31.965043 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:31.965161 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:31.965405 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:31.965408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:31.965601 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:31.965695 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:31.965701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:31.966416 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:31.966420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:31.966981 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:31.966989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:31.967407 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:31.967410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:31.967491 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:31.967931 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:31.967942 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:31.969791 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":78,"threads_started":1}
W20260926 13:45:31.970016 22144 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:31.964915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:44835 (local address 127.12.170.126:40817)
0926 13:45:31.965032 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:31.965037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:31.965107 (+ 70us) server_negotiation.cc:407] Connection header received
0926 13:45:31.965251 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:31.965254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:31.965301 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:31.965366 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:31.965800 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:31.966320 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:31.967088 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:31.967313 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:31.967394 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:31.969905 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.67:44835: BlockingRecv error: recv got EOF from 127.12.170.67:44835 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 13:45:32 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:32 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:32.612275 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:32.607140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.65:38245)
0926 13:45:32.607394 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:32.607408 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:32.607490 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:32.607704 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:32.607706 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:32.607872 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:32.607963 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:32.607968 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:32.608631 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:32.608633 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:32.609319 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:32.609332 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:32.609828 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:32.609833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:32.609923 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:32.610372 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:32.610383 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:32.612144 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":84,"threads_started":1}
W20260926 13:45:32.612422 22148 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:32.607240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:38245 (local address 127.12.170.126:40817)
0926 13:45:32.607457 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:32.607461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:32.607473 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:45:32.607551 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:32.607553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:32.607592 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:32.607685 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:32.608046 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:32.608552 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:32.609435 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:32.609704 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:32.609803 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:32.612292 (+ 2489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.65:38245: BlockingRecv error: recv got EOF from 127.12.170.65:38245 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":75,"threads_started":1}
Sep 26 13:45:32 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:32 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:32.794626 22149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:32.789223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.66:56635)
0926 13:45:32.789596 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:32.789611 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:32.789724 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:32.790050 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:32.790053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:32.790268 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:32.790380 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:32.790388 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:32.791176 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:32.791179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:32.791807 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:32.791816 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:32.792210 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:32.792213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:32.792278 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:32.792797 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:32.792811 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:32.794514 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":96,"threads_started":1}
W20260926 13:45:32.794811 22148 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:32.789248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:56635 (local address 127.12.170.126:40817)
0926 13:45:32.789399 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:32.789404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:32.789685 (+ 281us) server_negotiation.cc:407] Connection header received
0926 13:45:32.789870 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:32.789873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:32.789926 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:32.789991 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:32.790491 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:32.791055 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:32.791901 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:32.792114 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:32.792187 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:32.794687 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.66:56635: BlockingRecv error: recv got EOF from 127.12.170.66:56635 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 13:45:32 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:32 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:32.975514 22150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:32.970525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.67:43857)
0926 13:45:32.970771 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:32.970785 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:32.970881 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:32.971191 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:32.971194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:32.971376 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:32.971469 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:32.971475 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:32.972242 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:32.972246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:32.972824 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:32.972833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:32.973251 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:32.973254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:32.973340 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:32.973782 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:32.973792 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:32.975361 (+ 1569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":64,"threads_started":1}
W20260926 13:45:32.975598 22148 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:32.970649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:43857 (local address 127.12.170.126:40817)
0926 13:45:32.970793 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:32.970799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:32.970840 (+ 41us) server_negotiation.cc:407] Connection header received
0926 13:45:32.970994 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:32.970997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:32.971044 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:32.971117 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:32.971582 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:32.972135 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:32.972939 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:32.973153 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:32.973237 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:32.975496 (+ 2259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.67:43857: BlockingRecv error: recv got EOF from 127.12.170.67:43857 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 13:45:33 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:33 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:33.623215 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:33.613030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.65:60607)
0926 13:45:33.613306 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:33.613320 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:33.613406 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:33.613665 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:33.613693 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:33.613862 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:33.613957 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:33.613962 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:33.614649 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:33.614651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:33.615219 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:33.615226 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:33.615600 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:33.615603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:33.615707 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:33.621388 (+ 5681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:33.621411 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:33.623101 (+ 1690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":101,"threads_started":1}
W20260926 13:45:33.623397 22152 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:33.613105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:60607 (local address 127.12.170.126:40817)
0926 13:45:33.613344 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:33.613348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:33.613365 (+ 17us) server_negotiation.cc:407] Connection header received
0926 13:45:33.613485 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:33.613488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:33.613531 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:33.613613 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:33.614049 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:33.614572 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:33.615303 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:33.615516 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:33.615588 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:33.623247 (+ 7659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.65:60607: BlockingRecv error: recv got EOF from 127.12.170.65:60607 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":61,"threads_started":1}
Sep 26 13:45:33 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:33 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:33.800189 22153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:33.795327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.66:49885)
0926 13:45:33.795580 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:33.795594 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:33.795732 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:33.795934 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:33.795936 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:33.796095 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:33.796187 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:33.796192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:33.796869 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:33.796872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:33.797421 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:33.797429 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:33.797805 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:33.797807 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:33.797864 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:33.798315 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:33.798326 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:33.800055 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":65,"threads_started":1}
W20260926 13:45:33.800323 22152 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:33.795469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:49885 (local address 127.12.170.126:40817)
0926 13:45:33.795600 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:33.795604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:33.795660 (+ 56us) server_negotiation.cc:407] Connection header received
0926 13:45:33.795807 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:33.795810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:33.795851 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:33.795910 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:33.796272 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:33.796782 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:33.797511 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:33.797726 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:33.797797 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:33.800212 (+ 2415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.66:49885: BlockingRecv error: recv got EOF from 127.12.170.66:49885 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 13:45:33 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:33 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:33.981496 22154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:33.976344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.67:48309)
0926 13:45:33.976574 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:33.976586 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:33.976693 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:33.976890 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:33.976892 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:33.977053 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:33.977146 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:33.977152 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:33.977824 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:33.977827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:33.978371 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:33.978380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:33.978839 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:33.978841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:33.978904 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:33.979399 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:33.979412 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:33.981373 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":68,"threads_started":1}
W20260926 13:45:33.981607 22152 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:33.976385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:48309 (local address 127.12.170.126:40817)
0926 13:45:33.976502 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:33.976505 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:33.976642 (+ 137us) server_negotiation.cc:407] Connection header received
0926 13:45:33.976758 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:33.976760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:33.976803 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:33.976863 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:33.977233 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:33.977733 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:33.978485 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:33.978731 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:33.978822 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:33.981509 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.67:48309: BlockingRecv error: recv got EOF from 127.12.170.67:48309 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 13:45:34 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:34 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:34.628750 22155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:34.623888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.65:34409)
0926 13:45:34.624131 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:34.624144 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:34.624231 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:34.624443 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:34.624445 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:34.624617 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:34.624736 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:34.624742 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:34.625446 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:34.625449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:34.626034 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:34.626043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:34.626481 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:34.626483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:34.626538 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:34.626987 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:34.626997 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:34.628586 (+ 1589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
W20260926 13:45:34.628856 22156 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:34.623965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:34409 (local address 127.12.170.126:40817)
0926 13:45:34.624190 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:34.624194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:34.624206 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:45:34.624303 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:34.624304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:34.624345 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:34.624421 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:34.624834 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:34.625362 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:34.626157 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:34.626388 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:34.626466 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:34.628704 (+ 2238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.65:34409: BlockingRecv error: recv got EOF from 127.12.170.65:34409 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":84,"threads_started":1}
Sep 26 13:45:34 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:34 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:34.805656 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:34.800843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.66:38959)
0926 13:45:34.801077 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:34.801090 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:34.801175 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:34.801382 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:34.801384 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:34.801561 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:34.801652 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:34.801657 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:34.802376 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:34.802379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:34.802955 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:34.802962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:34.803306 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:34.803308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:34.803362 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:34.803828 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:34.803838 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:34.805528 (+ 1690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":66,"threads_started":1}
W20260926 13:45:34.805730 22156 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:34.800981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:38959 (local address 127.12.170.126:40817)
0926 13:45:34.801095 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:34.801098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:34.801136 (+ 38us) server_negotiation.cc:407] Connection header received
0926 13:45:34.801242 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:34.801244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:34.801294 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:34.801359 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:34.801775 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:34.802299 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:34.803033 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:34.803234 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:34.803308 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:34.805643 (+ 2335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.66:38959: BlockingRecv error: recv got EOF from 127.12.170.66:38959 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 13:45:34 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:34 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:34.987233 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:34.982220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.67:55269)
0926 13:45:34.982488 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:34.982503 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:34.982602 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:34.982844 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:34.982847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:34.983035 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:34.983123 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:34.983129 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:34.983896 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:34.983899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:34.984468 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:34.984477 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:34.984893 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:34.984896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:34.984976 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:34.985425 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:34.985435 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:34.987110 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":74,"threads_started":1}
W20260926 13:45:34.987347 22156 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:34.982408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:55269 (local address 127.12.170.126:40817)
0926 13:45:34.982543 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:34.982547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:34.982560 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:45:34.982692 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:34.982695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:34.982744 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:34.982806 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:34.983239 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:34.983792 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:34.984576 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:34.984797 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:34.984881 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:34.987248 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.67:55269: BlockingRecv error: recv got EOF from 127.12.170.67:55269 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 13:45:35 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:35 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:35.634394 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:35.629425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.65:52137)
0926 13:45:35.629690 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:35.629705 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:35.629812 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:35.630084 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:35.630086 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:35.630260 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:35.630356 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:35.630361 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:35.631118 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:35.631120 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:35.631691 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:35.631700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:35.632084 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:35.632087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:35.632160 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:35.632665 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:35.632677 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:35.634283 (+ 1606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":85,"threads_started":1}
W20260926 13:45:35.634507 22160 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:35.629471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:52137 (local address 127.12.170.126:40817)
0926 13:45:35.629756 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:35.629760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:35.629775 (+ 15us) server_negotiation.cc:407] Connection header received
0926 13:45:35.629900 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:35.629902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:35.629970 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:35.630066 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:35.630442 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:35.631035 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:35.631779 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:35.632001 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:35.632075 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:35.634400 (+ 2325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.65:52137: BlockingRecv error: recv got EOF from 127.12.170.65:52137 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":71,"threads_started":1}
Sep 26 13:45:35 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:35 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:35.811131 22161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:35.806354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.66:33555)
0926 13:45:35.806610 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:35.806622 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:35.806732 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:35.806928 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:35.806930 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:35.807100 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:35.807208 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:35.807214 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:35.807967 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:35.807970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:35.808529 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:35.808537 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:35.808899 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:35.808902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:35.808959 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:35.809408 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:35.809418 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:35.811018 (+ 1600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":91,"threads_started":1}
W20260926 13:45:35.811292 22160 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:35.806414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:33555 (local address 127.12.170.126:40817)
0926 13:45:35.806545 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:35.806550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:35.806687 (+ 137us) server_negotiation.cc:407] Connection header received
0926 13:45:35.806796 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:35.806798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:35.806841 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:35.806902 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:35.807298 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:35.807871 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:35.808608 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:35.808820 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:35.808889 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:35.811167 (+ 2278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.66:33555: BlockingRecv error: recv got EOF from 127.12.170.66:33555 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 13:45:35 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:35 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:35.993294 22162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:35.987945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.67:38191)
0926 13:45:35.988219 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:35.988235 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:35.988352 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:35.988621 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:35.988624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:35.988838 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:35.988934 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:35.988941 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:35.989687 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:35.989691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:35.990264 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:35.990273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:35.990722 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:35.990727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:35.990815 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:35.991305 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:35.991317 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:35.993120 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":80,"threads_started":1}
W20260926 13:45:35.993376 22160 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:35.988049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:38191 (local address 127.12.170.126:40817)
0926 13:45:35.988209 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:35.988214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:35.988309 (+ 95us) server_negotiation.cc:407] Connection header received
0926 13:45:35.988462 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:35.988466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:35.988517 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:35.988586 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:35.989052 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:35.989578 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:35.990382 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:35.990610 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:35.990711 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:35.993266 (+ 2555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.67:38191: BlockingRecv error: recv got EOF from 127.12.170.67:38191 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 13:45:36 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:36 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:36.640115 22163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:36.635051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.65:47559)
0926 13:45:36.635341 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:36.635356 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:36.635457 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:36.635679 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:36.635681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:36.635880 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:36.635974 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:36.635979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:36.636656 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:36.636658 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:36.637225 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:36.637232 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:36.637668 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:36.637670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:36.637749 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:36.638200 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:36.638211 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:36.640000 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":67,"threads_started":1}
W20260926 13:45:36.640259 22164 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:36.635115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:47559 (local address 127.12.170.126:40817)
0926 13:45:36.635351 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:36.635354 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:36.635421 (+ 67us) server_negotiation.cc:407] Connection header received
0926 13:45:36.635524 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:36.635526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:36.635567 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:36.635636 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:36.636059 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:36.636577 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:36.637343 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:36.637575 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:36.637656 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:36.640122 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.65:47559: BlockingRecv error: recv got EOF from 127.12.170.65:47559 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":79,"threads_started":1}
Sep 26 13:45:36 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:36 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:36.816927 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:36.811843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.66:47695)
0926 13:45:36.812102 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:36.812114 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:36.812221 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:36.812464 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:36.812467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:36.812651 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:36.812758 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:36.812765 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:36.813622 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:36.813625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:36.814197 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:36.814207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:36.814605 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:36.814607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:36.814669 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:36.815138 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:36.815148 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:36.816815 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":64,"threads_started":1}
W20260926 13:45:36.817054 22164 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:36.811917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:47695 (local address 127.12.170.126:40817)
0926 13:45:36.812042 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:36.812045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:36.812205 (+ 160us) server_negotiation.cc:407] Connection header received
0926 13:45:36.812287 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:36.812289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:36.812339 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:36.812420 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:36.812852 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:36.813505 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:36.814295 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:36.814515 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:36.814596 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:36.816950 (+ 2354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.66:47695: BlockingRecv error: recv got EOF from 127.12.170.66:47695 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 13:45:36 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:36 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:36.999305 22166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:36.994127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.67:60439)
0926 13:45:36.994474 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:36.994485 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:36.994588 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:36.994832 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:36.994835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:36.995020 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:36.995110 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:36.995115 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:36.995906 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:36.995910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:36.996461 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:36.996468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:36.996884 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:36.996888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:36.996968 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:36.997425 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:36.997435 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:36.999181 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":65,"threads_started":1}
W20260926 13:45:36.999399 22164 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:36.994206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:60439 (local address 127.12.170.126:40817)
0926 13:45:36.994356 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:36.994361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:36.994557 (+ 196us) server_negotiation.cc:407] Connection header received
0926 13:45:36.994680 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:36.994683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:36.994728 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:36.994790 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:36.995218 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:36.995807 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:36.996562 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:36.996787 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:36.996870 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:36.999296 (+ 2426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.67:60439: BlockingRecv error: recv got EOF from 127.12.170.67:60439 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 13:45:37 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:37 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:37.646103 22167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:37.640839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.65:36915)
0926 13:45:37.641072 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:37.641086 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:37.641180 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:37.641400 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:37.641402 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:37.641574 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:37.641685 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:37.641691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:37.642532 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:37.642536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:37.643236 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:37.643247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:37.643706 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:37.643709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:37.643786 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:37.644277 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:37.644290 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:37.645990 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":75,"threads_started":1}
W20260926 13:45:37.646251 22168 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:37.640967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:36915 (local address 127.12.170.126:40817)
0926 13:45:37.641175 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:37.641179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:37.641191 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:45:37.641257 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:37.641259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:37.641304 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:37.641381 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:37.641812 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:37.642433 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:37.643357 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:37.643580 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:37.643705 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:37.646116 (+ 2411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.65:36915: BlockingRecv error: recv got EOF from 127.12.170.65:36915 (error 108)
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":71,"threads_started":1}
Sep 26 13:45:37 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:37 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:37.823010 22169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:37.817682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.66:38393)
0926 13:45:37.817952 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:37.817969 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:37.818081 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:37.818311 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:37.818314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:37.818507 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:37.818597 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:37.818603 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:37.819527 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:37.819530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:37.820118 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:37.820127 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:37.820572 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:37.820576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:37.820656 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:37.821142 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:37.821162 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:37.822867 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":89,"threads_started":1}
W20260926 13:45:37.823098 22168 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:37.817787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:38393 (local address 127.12.170.126:40817)
0926 13:45:37.817953 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:37.817958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:37.818045 (+ 87us) server_negotiation.cc:407] Connection header received
0926 13:45:37.818168 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:37.818171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:37.818219 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:37.818281 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:37.818746 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:37.819433 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:37.820239 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:37.820472 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:37.820568 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:37.823001 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.66:38393: BlockingRecv error: recv got EOF from 127.12.170.66:38393 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 26 13:45:38 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:38 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:38.005393 22170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:37.999989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.67:57629)
0926 13:45:38.000243 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:38.000257 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:38.000360 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:38.000582 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:38.000585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:38.000796 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:38.000897 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:38.000904 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:38.001639 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:38.001642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:38.002232 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:38.002243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:38.002809 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:38.002815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:38.002908 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:38.003424 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:38.003437 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:38.005272 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":82,"threads_started":1}
W20260926 13:45:38.005523 22168 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:38.000072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:57629 (local address 127.12.170.126:40817)
0926 13:45:38.000230 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:38.000235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:38.000320 (+ 85us) server_negotiation.cc:407] Connection header received
0926 13:45:38.000430 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:38.000432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:38.000477 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:38.000562 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:38.001000 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:38.001533 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:38.002346 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:38.002658 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:38.002793 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:38.005401 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.67:57629: BlockingRecv error: recv got EOF from 127.12.170.67:57629 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 13:45:38 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:38 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:38.655202 22171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:38.646764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.65:56945)
0926 13:45:38.647003 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:38.647015 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:38.647097 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:38.647324 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:38.647326 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:38.647488 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:38.647579 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:38.647584 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:38.648324 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:38.648327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:38.648882 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:38.648889 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:38.649274 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:38.649277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:38.649334 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:38.653396 (+ 4062us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:38.653408 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:38.655081 (+ 1673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":67,"threads_started":1}
W20260926 13:45:38.655332 22172 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:38.646814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:56945 (local address 127.12.170.126:40817)
0926 13:45:38.647041 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:38.647045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:38.647057 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:45:38.647182 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:38.647184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:38.647226 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:38.647301 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:38.647684 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:38.648234 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:38.648966 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:38.649195 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:38.649265 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:38.655214 (+ 5949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.65:56945: BlockingRecv error: recv got EOF from 127.12.170.65:56945 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":81,"threads_started":1}
Sep 26 13:45:38 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:38 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:38.828868 22173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:38.823804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.66:41123)
0926 13:45:38.824044 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:38.824059 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:38.824176 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:38.824404 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:38.824407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:38.824589 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:38.824686 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:38.824691 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:38.825411 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:38.825414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:38.825963 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:38.825971 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:38.826400 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:38.826404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:38.826485 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:38.826942 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:38.826954 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:38.828702 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":81,"threads_started":1}
W20260926 13:45:38.828927 22172 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:38.823859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:41123 (local address 127.12.170.126:40817)
0926 13:45:38.823987 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:38.823991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:38.824122 (+ 131us) server_negotiation.cc:407] Connection header received
0926 13:45:38.824258 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:38.824261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:38.824309 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:38.824371 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:38.824790 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:38.825314 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:38.826065 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:38.826303 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:38.826387 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:38.828830 (+ 2443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.66:41123: BlockingRecv error: recv got EOF from 127.12.170.66:41123 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 13:45:39 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:39 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:39.010957 22174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:39.006033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.67:38515)
0926 13:45:39.006289 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:39.006303 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:39.006394 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:39.006591 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:39.006593 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:39.006758 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:39.006852 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:39.006859 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:39.007547 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:39.007550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:39.008131 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:39.008141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:39.008502 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:39.008504 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:39.008562 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:39.008995 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:39.009007 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:39.010814 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":86,"threads_started":1}
W20260926 13:45:39.011101 22172 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:39.006100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:38515 (local address 127.12.170.126:40817)
0926 13:45:39.006228 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:39.006232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:39.006359 (+ 127us) server_negotiation.cc:407] Connection header received
0926 13:45:39.006460 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:39.006462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:39.006505 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:39.006567 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:39.006941 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:39.007456 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:39.008225 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:39.008429 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:39.008493 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:39.010978 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.67:38515: BlockingRecv error: recv got EOF from 127.12.170.67:38515 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 13:45:39 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:39 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:39.660869 22175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:39.655895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.65:34275)
0926 13:45:39.656134 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:39.656147 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:39.656241 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:39.656453 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:39.656455 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:39.656658 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:39.656800 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:39.656809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:39.657526 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:39.657528 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:39.658133 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:39.658143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:39.658553 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:39.658555 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:39.658619 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:39.659066 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:39.659076 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:39.660746 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":67,"threads_started":1}
W20260926 13:45:39.661062 22176 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:39.655948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:34275 (local address 127.12.170.126:40817)
0926 13:45:39.656190 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:39.656193 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:39.656205 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:45:39.656315 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:39.656317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:39.656358 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:39.656426 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:39.656898 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:39.657426 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:39.658237 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:39.658459 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:39.658527 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:39.660898 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.65:34275: BlockingRecv error: recv got EOF from 127.12.170.65:34275 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
Sep 26 13:45:39 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:39 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:39.834535 22177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:39.829693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.66:47033)
0926 13:45:39.829907 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:39.829919 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:39.830008 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:39.830222 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:39.830224 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:39.830404 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:39.830496 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:39.830502 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:39.831142 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:39.831144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:39.831754 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:39.831762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:39.832111 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:39.832113 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:39.832173 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:39.832627 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:39.832638 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:39.834391 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":77,"threads_started":1}
W20260926 13:45:39.834614 22176 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:39.829727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:47033 (local address 127.12.170.126:40817)
0926 13:45:39.829881 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:39.829886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:39.829967 (+ 81us) server_negotiation.cc:407] Connection header received
0926 13:45:39.830068 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:39.830070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:39.830112 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:39.830210 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:39.830567 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:39.831069 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:39.831837 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:39.832039 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:39.832105 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:39.834519 (+ 2414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.66:47033: BlockingRecv error: recv got EOF from 127.12.170.66:47033 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 13:45:40 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:40 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:40.016923 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:40.011811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.67:42403)
0926 13:45:40.012073 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:40.012088 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:40.012216 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:40.012455 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:40.012458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:40.012681 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:40.012802 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:40.012811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:40.013531 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:40.013534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:40.014101 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:40.014109 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:40.014521 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:40.014524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:40.014603 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:40.015047 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:40.015058 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:40.016766 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
W20260926 13:45:40.016994 22176 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:40.011976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:42403 (local address 127.12.170.126:40817)
0926 13:45:40.012098 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:40.012103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:40.012167 (+ 64us) server_negotiation.cc:407] Connection header received
0926 13:45:40.012303 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:40.012306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:40.012352 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:40.012414 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:40.012910 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:40.013428 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:40.014211 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:40.014427 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:40.014508 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:40.016895 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.67:42403: BlockingRecv error: recv got EOF from 127.12.170.67:42403 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 13:45:40 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:40 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:40.666847 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:40.661537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.65:32969)
0926 13:45:40.661827 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:40.661840 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:40.661931 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:40.662224 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:40.662227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:40.662406 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:40.662513 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:40.662519 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:40.663250 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:40.663253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:40.663845 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:40.663854 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:40.664257 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:40.664260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:40.664325 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:40.664910 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:40.664925 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:40.666704 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":116,"threads_started":1}
W20260926 13:45:40.666949 22180 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:40.661591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:32969 (local address 127.12.170.126:40817)
0926 13:45:40.661850 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:40.661853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:40.661900 (+ 47us) server_negotiation.cc:407] Connection header received
0926 13:45:40.662032 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:40.662034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:40.662080 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:40.662170 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:40.662611 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:40.663149 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:40.663938 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:40.664170 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:40.664242 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:40.666835 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.65:32969: BlockingRecv error: recv got EOF from 127.12.170.65:32969 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":74,"threads_started":1}
Sep 26 13:45:40 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:40 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:40.840150 22181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:40.835355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.66:41887)
0926 13:45:40.835587 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:40.835601 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:40.835721 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:40.835918 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:40.835920 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:40.836086 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:40.836182 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:40.836188 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:40.836943 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:40.836945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:40.837493 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:40.837500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:40.837869 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:40.837871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:40.837927 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:40.838382 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:40.838394 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:40.840038 (+ 1644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":65,"threads_started":1}
W20260926 13:45:40.840298 22180 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:40.835434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:41887 (local address 127.12.170.126:40817)
0926 13:45:40.835589 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:40.835594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:40.835665 (+ 71us) server_negotiation.cc:407] Connection header received
0926 13:45:40.835789 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:40.835791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:40.835834 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:40.835896 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:40.836283 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:40.836851 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:40.837573 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:40.837792 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:40.837861 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:40.840187 (+ 2326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.66:41887: BlockingRecv error: recv got EOF from 127.12.170.66:41887 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 13:45:41 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:41 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:41.022720 22182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:41.017635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.67:35087)
0926 13:45:41.017916 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:41.017929 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:41.018034 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:41.018265 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:41.018267 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:41.018452 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:41.018544 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:41.018550 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:41.019296 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:41.019299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:41.019953 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:41.019962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:41.020370 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:41.020374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:41.020452 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:41.020904 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:41.020916 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:41.022608 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":92,"threads_started":1}
W20260926 13:45:41.022841 22180 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:41.017695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:35087 (local address 127.12.170.126:40817)
0926 13:45:41.017871 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:41.017875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:41.018000 (+ 125us) server_negotiation.cc:407] Connection header received
0926 13:45:41.018118 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:41.018121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:41.018170 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:41.018232 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:41.018685 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:41.019195 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:41.020054 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:41.020275 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:41.020358 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:41.022729 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.67:35087: BlockingRecv error: recv got EOF from 127.12.170.67:35087 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 26 13:45:41 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:41 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:41.678188 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:41.667548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.65:40923)
0926 13:45:41.667808 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:41.667820 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:41.667904 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:41.668108 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:41.668110 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:41.668288 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:41.668383 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:41.668388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:41.669095 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:41.669097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:41.669652 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:41.669659 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:41.670101 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:41.670104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:41.670186 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:41.676326 (+ 6140us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:41.676339 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:41.678066 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":85,"threads_started":1}
W20260926 13:45:41.678315 22184 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:41.667599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:40923 (local address 127.12.170.126:40817)
0926 13:45:41.667855 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:41.667858 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:41.667871 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:45:41.667974 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:41.667976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:41.668020 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:41.668088 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:41.668475 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:41.669016 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:41.669761 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:41.669991 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:41.670073 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:41.678200 (+ 8127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.65:40923: BlockingRecv error: recv got EOF from 127.12.170.65:40923 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":105,"threads_started":1}
Sep 26 13:45:41 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:41 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:41.845875 22185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:41.840804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.66:51583)
0926 13:45:41.841033 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:41.841045 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:41.841151 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:41.841360 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:41.841362 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:41.841526 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:41.841623 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:41.841628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:41.842390 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:41.842392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:41.842942 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:41.842950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:41.843317 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:41.843320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:41.843374 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:41.843917 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:41.843928 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:41.845761 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":75,"threads_started":1}
W20260926 13:45:41.845976 22184 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:41.840863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:51583 (local address 127.12.170.126:40817)
0926 13:45:41.840987 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:41.840991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:41.841098 (+ 107us) server_negotiation.cc:407] Connection header received
0926 13:45:41.841225 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:41.841227 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:41.841272 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:41.841338 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:41.841699 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:41.842301 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:41.843026 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:41.843244 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:41.843311 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:41.845880 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.66:51583: BlockingRecv error: recv got EOF from 127.12.170.66:51583 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 13:45:42 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:42 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:42.028915 22186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:42.023454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.67:44011)
0926 13:45:42.023762 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:42.023777 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:42.023893 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:42.024175 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:42.024178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:42.024397 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:42.024496 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:42.024502 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:42.025213 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:42.025216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:42.025897 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:42.025908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:42.026388 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:42.026391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:42.026454 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:42.026920 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:42.026932 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:42.028779 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":95,"threads_started":1}
W20260926 13:45:42.029012 22184 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:42.023482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:44011 (local address 127.12.170.126:40817)
0926 13:45:42.023600 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:42.023604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:42.023858 (+ 254us) server_negotiation.cc:407] Connection header received
0926 13:45:42.023987 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:42.023990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:42.024040 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:42.024127 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:42.024593 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:42.025116 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:42.026033 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:42.026270 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:42.026356 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:42.028907 (+ 2551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.67:44011: BlockingRecv error: recv got EOF from 127.12.170.67:44011 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 13:45:42 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:42 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:42.683928 22187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:42.678839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.65:55053)
0926 13:45:42.679109 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:42.679123 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:42.679212 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:42.679463 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:42.679466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:42.679679 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:42.679786 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:42.679793 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:42.680482 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:42.680485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:42.681038 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:42.681045 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:42.681431 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:42.681434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:42.681494 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:42.681994 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:42.682007 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:42.683783 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":68,"threads_started":1}
W20260926 13:45:42.684015 22188 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:42.678900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:55053 (local address 127.12.170.126:40817)
0926 13:45:42.679131 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:42.679141 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:42.679166 (+ 25us) server_negotiation.cc:407] Connection header received
0926 13:45:42.679282 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:42.679284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:42.679324 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:42.679395 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:42.679879 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:42.680398 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:42.681121 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:42.681354 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:42.681422 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:42.683904 (+ 2482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.65:55053: BlockingRecv error: recv got EOF from 127.12.170.65:55053 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":73,"threads_started":1}
Sep 26 13:45:42 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:42 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:42.851801 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:42.846559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.66:43671)
0926 13:45:42.846841 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:42.846856 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:42.846963 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:42.847196 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:42.847199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:42.847395 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:42.847483 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:42.847489 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:42.848259 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:42.848262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:42.848837 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:42.848845 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:42.849254 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:42.849257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:42.849335 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:42.849841 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:42.849852 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:42.851675 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":89,"threads_started":1}
W20260926 13:45:42.851915 22188 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:42.846714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:43671 (local address 127.12.170.126:40817)
0926 13:45:42.846842 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:42.846846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:42.846909 (+ 63us) server_negotiation.cc:407] Connection header received
0926 13:45:42.847045 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:42.847048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:42.847091 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:42.847154 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:42.847587 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:42.848160 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:42.848941 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:42.849161 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:42.849243 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:42.851818 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.66:43671: BlockingRecv error: recv got EOF from 127.12.170.66:43671 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 13:45:43 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:43 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:43.034812 22190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:43.029574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.67:35669)
0926 13:45:43.029868 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:43.029880 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:43.029997 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:43.030241 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:43.030244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:43.030436 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:43.030526 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:43.030532 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:43.031372 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:43.031375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:43.032000 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:43.032009 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:43.032414 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:43.032416 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:43.032471 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:43.032968 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:43.032979 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:43.034691 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":112,"threads_started":1}
W20260926 13:45:43.034950 22188 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:43.029629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:35669 (local address 127.12.170.126:40817)
0926 13:45:43.029759 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:43.029764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:43.029944 (+ 180us) server_negotiation.cc:407] Connection header received
0926 13:45:43.030078 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:43.030081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:43.030143 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:43.030207 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:43.030671 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:43.031264 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:43.032108 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:43.032331 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:43.032413 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:43.034825 (+ 2412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.67:35669: BlockingRecv error: recv got EOF from 127.12.170.67:35669 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 13:45:43 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:43 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:43.693794 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:43.684686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.65:45079)
0926 13:45:43.684981 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:43.684996 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:43.685098 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:43.685339 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:43.685341 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:43.685519 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:43.685621 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:43.685627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:43.686351 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:43.686354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:43.686916 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:43.686924 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:43.687365 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:43.687368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:43.687435 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:43.691967 (+ 4532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:43.691979 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:43.693679 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":75,"threads_started":1}
W20260926 13:45:43.693981 22192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:43.684747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:45079 (local address 127.12.170.126:40817)
0926 13:45:43.685038 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:43.685042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:43.685054 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:45:43.685175 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:43.685177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:43.685217 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:43.685287 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:43.685711 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:43.686238 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:43.687019 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:43.687250 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:43.687321 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:43.693840 (+ 6519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.65:45079: BlockingRecv error: recv got EOF from 127.12.170.65:45079 (error 108)
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":85,"threads_started":1}
Sep 26 13:45:43 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:43 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:43.857610 22193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:43.852471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.66:35607)
0926 13:45:43.852760 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:43.852775 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:43.852881 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:43.853102 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:43.853104 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:43.853282 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:43.853384 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:43.853391 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:43.854041 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:43.854043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:43.854632 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:43.854640 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:43.855005 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:43.855007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:43.855102 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:43.855555 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:43.855569 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:43.857490 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":108,"threads_started":1}
W20260926 13:45:43.857810 22192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:43.852533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:35607 (local address 127.12.170.126:40817)
0926 13:45:43.852654 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:43.852658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:43.852838 (+ 180us) server_negotiation.cc:407] Connection header received
0926 13:45:43.852955 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:43.852957 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:43.852996 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:43.853059 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:43.853466 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:43.853960 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:43.854726 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:43.854928 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:43.854991 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:43.857689 (+ 2698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.66:35607: BlockingRecv error: recv got EOF from 127.12.170.66:35607 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 26 13:45:44 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:44 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.67@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:44.040894 22194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:44.035525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.67:40545)
0926 13:45:44.035810 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:44.035826 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:44.035944 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:44.036372 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:44.036376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:44.036622 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:44.036762 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:44.036772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:44.037511 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:44.037513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:44.038108 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:44.038117 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:44.038533 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:44.038536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:44.038628 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:44.039099 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:44.039110 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:44.040751 (+ 1641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":107,"threads_started":1}
W20260926 13:45:44.041015 22192 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:44.035593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:40545 (local address 127.12.170.126:40817)
0926 13:45:44.035732 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:44.035737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:44.035903 (+ 166us) server_negotiation.cc:407] Connection header received
0926 13:45:44.036036 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:44.036039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:44.036097 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:44.036171 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:44.036893 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:44.037412 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:44.038220 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:44.038436 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:44.038510 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:44.040918 (+ 2408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.67:40545: BlockingRecv error: recv got EOF from 127.12.170.67:40545 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 13:45:44 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:44 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.65@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:44.699877 22195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:44.694655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.65:38589)
0926 13:45:44.694936 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:44.694949 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:44.695041 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:44.695255 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:44.695257 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:44.695434 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:44.695526 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:44.695530 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:44.696231 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:44.696233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:44.696796 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:44.696803 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:44.697247 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:44.697252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:44.697323 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:44.697906 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:44.697921 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:44.699721 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":71,"threads_started":1}
W20260926 13:45:44.699967 22196 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:44.694725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:38589 (local address 127.12.170.126:40817)
0926 13:45:44.694958 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:44.694961 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:44.694997 (+ 36us) server_negotiation.cc:407] Connection header received
0926 13:45:44.695113 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:44.695115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:44.695156 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:44.695227 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:44.695613 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:44.696146 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:44.696900 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:44.697137 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:44.697216 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:44.699848 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.65:38589: BlockingRecv error: recv got EOF from 127.12.170.65:38589 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":79,"threads_started":1}
Sep 26 13:45:44 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:45:44 dist-test-slave-hc1k krb5kdc[21138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.170.66@KRBTEST.COM for oryx/127.12.170.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:45:44.863336 22197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:44.858378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40817 (local address 127.12.170.66:40651)
0926 13:45:44.858617 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:44.858630 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:44.858716 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:44.858921 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:44.858923 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:44.859083 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:44.859188 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:44.859193 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:44.859981 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:44.859984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:44.860544 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:44.860552 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:44.860982 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:44.860984 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:44.861043 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:44.861525 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:44.861535 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:44.863226 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.170.126:40817: Server oryx/127.12.170.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":80,"threads_started":1}
W20260926 13:45:44.863462 22196 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:44.858431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:40651 (local address 127.12.170.126:40817)
0926 13:45:44.858595 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:44.858601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:44.858687 (+ 86us) server_negotiation.cc:407] Connection header received
0926 13:45:44.858785 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:44.858787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:44.858832 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:44.858901 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:44.859283 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:44.859862 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:44.860659 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:44.860889 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:44.860983 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:44.863350 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.170.66:40651: BlockingRecv error: recv got EOF from 127.12.170.66:40651 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 13:45:44.945875 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 21721
I20260926 13:45:44.950282 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 21873
I20260926 13:45:44.954653 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 22005
I20260926 13:45:44.958895 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 21651
2026-09-26T13:45:44Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17706 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T13:45:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:45:44Z Disabled control of system clock
I20260926 13:45:44.985234 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:42457
--webserver_interface=127.12.170.126
--webserver_port=0
--builtin_ntp_servers=127.12.170.84:46679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:42457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 13:45:45.086431 22201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:45.086634 22201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:45.086663 22201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:45.090407 22201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:45:45.090446 22201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:45.090461 22201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:45.090474 22201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:45:45.090487 22201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:45:45.094901 22201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:46679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:42457
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:42457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:45.095746 22201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:45.096905 22201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:45.104488 22209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:45.104512 22206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:45.104646 22201 server_base.cc:1034] running on GCE node
W20260926 13:45:45.104605 22207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:45.105211 22201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:45.106115 22201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:45.107267 22201 hybrid_clock.cc:648] HybridClock initialized: now 1790430345107252 us; error 24 us; skew 500 ppm
I20260926 13:45:45.108930 22201 webserver.cc:460] Webserver started at http://127.12.170.126:34447/ using document root <none> and password file <none>
I20260926 13:45:45.109440 22201 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:45.109498 22201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:45.109685 22201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:45.111490 22201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "f2929f63e8fa4e7e9e30ff5bec6f6227"
format_stamp: "Formatted at 2026-09-26 13:45:45 on dist-test-slave-hc1k"
server_key: "ccb5b45c2afa0901b24081b6a0cea438"
server_key_iv: "0b165eb223196aae9037ccc374ce70c9"
server_key_version: "encryptionkey@0"
I20260926 13:45:45.112026 22201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "f2929f63e8fa4e7e9e30ff5bec6f6227"
format_stamp: "Formatted at 2026-09-26 13:45:45 on dist-test-slave-hc1k"
server_key: "ccb5b45c2afa0901b24081b6a0cea438"
server_key_iv: "0b165eb223196aae9037ccc374ce70c9"
server_key_version: "encryptionkey@0"
I20260926 13:45:45.115190 22201 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:45:45.117280 22215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:45.118376 22201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:45.118477 22201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "f2929f63e8fa4e7e9e30ff5bec6f6227"
format_stamp: "Formatted at 2026-09-26 13:45:45 on dist-test-slave-hc1k"
server_key: "ccb5b45c2afa0901b24081b6a0cea438"
server_key_iv: "0b165eb223196aae9037ccc374ce70c9"
server_key_version: "encryptionkey@0"
I20260926 13:45:45.118559 22201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:45.142982 22201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:45.143573 22201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:45.143761 22201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:45.149819 22201 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:42457
I20260926 13:45:45.149843 22267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:42457 every 8 connection(s)
I20260926 13:45:45.150651 22201 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:45:45.153635 22268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:45.158824 22268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2929f63e8fa4e7e9e30ff5bec6f6227: Bootstrap starting.
I20260926 13:45:45.159812 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 22201
I20260926 13:45:45.159925 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:45:45.160213 12969 external_mini_cluster.cc:1428] Setting key e69f9e7600d0232b986aab9c8ae48e12
I20260926 13:45:45.161441 22268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2929f63e8fa4e7e9e30ff5bec6f6227: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:45.162181 22268 log.cc:824] T 00000000000000000000000000000000 P f2929f63e8fa4e7e9e30ff5bec6f6227: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:45.163939 22268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2929f63e8fa4e7e9e30ff5bec6f6227: No bootstrap required, opened a new log
I20260926 13:45:45.167071 22268 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2929f63e8fa4e7e9e30ff5bec6f6227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2929f63e8fa4e7e9e30ff5bec6f6227" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 42457 } }
I20260926 13:45:45.167268 22268 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2929f63e8fa4e7e9e30ff5bec6f6227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:45.167322 22268 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2929f63e8fa4e7e9e30ff5bec6f6227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2929f63e8fa4e7e9e30ff5bec6f6227, State: Initialized, Role: FOLLOWER
I20260926 13:45:45.167809 22268 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2929f63e8fa4e7e9e30ff5bec6f6227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2929f63e8fa4e7e9e30ff5bec6f6227" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 42457 } }
I20260926 13:45:45.167954 22268 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2929f63e8fa4e7e9e30ff5bec6f6227 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:45:45.168012 22268 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2929f63e8fa4e7e9e30ff5bec6f6227 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:45:45.168111 22268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2929f63e8fa4e7e9e30ff5bec6f6227 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:45.168715 22271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:45.161279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60036 (local address 127.12.170.126:42457)
0926 13:45:45.161917 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:45.161930 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:45.161963 (+ 33us) server_negotiation.cc:407] Connection header received
0926 13:45:45.162590 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:45.162608 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:45.162925 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:45.163209 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:45.163592 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:45.164597 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:45.165447 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:45.165897 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:45.166035 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:45.166767 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:45.166772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:45.166783 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:45:45.167130 (+ 347us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:45.167182 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:45:45.167201 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:45.167281 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:45.167487 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 13:45:45.167791 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":129,"threads_started":1}
I20260926 13:45:45.169099 22268 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2929f63e8fa4e7e9e30ff5bec6f6227 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2929f63e8fa4e7e9e30ff5bec6f6227" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 42457 } }
I20260926 13:45:45.169392 22268 leader_election.cc:302] T 00000000000000000000000000000000 P f2929f63e8fa4e7e9e30ff5bec6f6227 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2929f63e8fa4e7e9e30ff5bec6f6227; no voters:
I20260926 13:45:45.169664 22268 leader_election.cc:288] T 00000000000000000000000000000000 P f2929f63e8fa4e7e9e30ff5bec6f6227 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:45:45.169734 22273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2929f63e8fa4e7e9e30ff5bec6f6227 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:45.169936 22273 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2929f63e8fa4e7e9e30ff5bec6f6227 [term 1 LEADER]: Becoming Leader. State: Replica: f2929f63e8fa4e7e9e30ff5bec6f6227, State: Running, Role: LEADER
I20260926 13:45:45.170264 22273 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2929f63e8fa4e7e9e30ff5bec6f6227 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2929f63e8fa4e7e9e30ff5bec6f6227" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 42457 } }
I20260926 13:45:45.170814 22268 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2929f63e8fa4e7e9e30ff5bec6f6227 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:45:45.171752 22274 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2929f63e8fa4e7e9e30ff5bec6f6227 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2929f63e8fa4e7e9e30ff5bec6f6227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2929f63e8fa4e7e9e30ff5bec6f6227" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 42457 } } }
I20260926 13:45:45.171864 22274 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2929f63e8fa4e7e9e30ff5bec6f6227 [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:45.172201 22282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:45:45.172173 22275 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2929f63e8fa4e7e9e30ff5bec6f6227 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2929f63e8fa4e7e9e30ff5bec6f6227. Latest consensus state: current_term: 1 leader_uuid: "f2929f63e8fa4e7e9e30ff5bec6f6227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2929f63e8fa4e7e9e30ff5bec6f6227" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 42457 } } }
I20260926 13:45:45.172344 22275 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2929f63e8fa4e7e9e30ff5bec6f6227 [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:45.174707 22282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:45:45.179174 22282 catalog_manager.cc:1347] Generated new cluster ID: 7327a45f46e644ca91867ff9a94fcad1
I20260926 13:45:45.179234 22282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:45:45.187603 22282 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:45:45.188690 22282 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:45:45.204557 22282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2929f63e8fa4e7e9e30ff5bec6f6227: Generated new TSK 0
I20260926 13:45:45.205166 22282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:45:45.210749 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:0
--local_ip_for_outbound_sockets=127.12.170.65
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:42457
--builtin_ntp_servers=127.12.170.84:46679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 13:45:45.315840 22292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:45.316073 22292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:45.316114 22292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:45.320025 22292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:45.320077 22292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:45.320152 22292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:45:45.324868 22292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:46679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:42457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:45.325781 22292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:45.327035 22292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:45.335736 22298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:45.335863 22292 server_base.cc:1034] running on GCE node
W20260926 13:45:45.335872 22297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:45.335768 22300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:45.336359 22292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:45.336887 22292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:45.338048 22292 hybrid_clock.cc:648] HybridClock initialized: now 1790430345338033 us; error 29 us; skew 500 ppm
I20260926 13:45:45.339684 22292 webserver.cc:460] Webserver started at http://127.12.170.65:37257/ using document root <none> and password file <none>
I20260926 13:45:45.340190 22292 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:45.340251 22292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:45.340417 22292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:45.342247 22292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/data/instance:
uuid: "51c72f01495a41948dc9ef564bd75f86"
format_stamp: "Formatted at 2026-09-26 13:45:45 on dist-test-slave-hc1k"
server_key: "f72b8a33634d29ee26412271a47a6b30"
server_key_iv: "99c3b17a9edecb8e6b85a9cd23bf6a9c"
server_key_version: "encryptionkey@0"
I20260926 13:45:45.342737 22292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance:
uuid: "51c72f01495a41948dc9ef564bd75f86"
format_stamp: "Formatted at 2026-09-26 13:45:45 on dist-test-slave-hc1k"
server_key: "f72b8a33634d29ee26412271a47a6b30"
server_key_iv: "99c3b17a9edecb8e6b85a9cd23bf6a9c"
server_key_version: "encryptionkey@0"
I20260926 13:45:45.345974 22292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:45:45.348140 22306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:45.349224 22292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:45:45.349318 22292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "51c72f01495a41948dc9ef564bd75f86"
format_stamp: "Formatted at 2026-09-26 13:45:45 on dist-test-slave-hc1k"
server_key: "f72b8a33634d29ee26412271a47a6b30"
server_key_iv: "99c3b17a9edecb8e6b85a9cd23bf6a9c"
server_key_version: "encryptionkey@0"
I20260926 13:45:45.349402 22292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:45.381361 22292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:45.382016 22292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:45.382192 22292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:45.382727 22292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:45.383599 22292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:45.383644 22292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:45.383719 22292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:45.383744 22292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:45.391546 22292 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:34205
I20260926 13:45:45.391577 22419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:34205 every 8 connection(s)
I20260926 13:45:45.392355 22292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:45:45.396633 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 22292
I20260926 13:45:45.396736 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance
I20260926 13:45:45.396987 12969 external_mini_cluster.cc:1428] Setting key dd01a019496703c40c6b085b8e50411a
I20260926 13:45:45.398705 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:0
--local_ip_for_outbound_sockets=127.12.170.66
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:42457
--builtin_ntp_servers=127.12.170.84:46679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 13:45:45.400976 22271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:45.393953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:54437 (local address 127.12.170.126:42457)
0926 13:45:45.394070 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:45.394074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:45.395079 (+ 1005us) server_negotiation.cc:407] Connection header received
0926 13:45:45.396539 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:45.396543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:45.396607 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:45.396679 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:45.397552 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:45.398080 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:45.399208 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:45.399427 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:45.399510 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:45.400522 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:45.400526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:45.400529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:45:45.400542 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:45.400561 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:45:45.400569 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:45.400652 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:45.400789 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 13:45:45.400844 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 13:45:45.401754 22422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:45.394291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:42457 (local address 127.12.170.65:54437)
0926 13:45:45.394898 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:45.394949 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:45.396336 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:45.396854 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:45.396861 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:45.396940 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:45.397418 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:45.397431 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:45.398188 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:45.398191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:45.399101 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:45.399110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:45.399550 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:45.399554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:45.399738 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:45.400291 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:45:45.400292 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:45.400328 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:45:45.400385 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:45.400663 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:45.400672 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:45.400677 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:45:45.400779 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 13:45:45.401026 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":96,"threads_started":1}
I20260926 13:45:45.402803 22420 heartbeater.cc:347] Connected to a master server at 127.12.170.126:42457
I20260926 13:45:45.403039 22420 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:45.403617 22420 heartbeater.cc:511] Master 127.12.170.126:42457 requested a full tablet report, sending...
I20260926 13:45:45.405354 22232 ts_manager.cc:194] Registered new tserver with Master: 51c72f01495a41948dc9ef564bd75f86 (127.12.170.65:34205)
I20260926 13:45:45.406883 22232 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.170.65:54437
W20260926 13:45:45.501525 22423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:45.501750 22423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:45.501783 22423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:45.505651 22423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:45.505693 22423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:45.505741 22423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:45:45.510188 22423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:46679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:42457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:45.511042 22423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:45.512282 22423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:45.520845 22428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:45.520892 22431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:45.520838 22429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:45.521135 22423 server_base.cc:1034] running on GCE node
I20260926 13:45:45.521492 22423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:45.521997 22423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:45.523138 22423 hybrid_clock.cc:648] HybridClock initialized: now 1790430345523111 us; error 45 us; skew 500 ppm
I20260926 13:45:45.524720 22423 webserver.cc:460] Webserver started at http://127.12.170.66:44985/ using document root <none> and password file <none>
I20260926 13:45:45.525209 22423 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:45.525266 22423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:45.525425 22423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:45.527228 22423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/data/instance:
uuid: "3e84e6cc58604da0b7a4236545da6df0"
format_stamp: "Formatted at 2026-09-26 13:45:45 on dist-test-slave-hc1k"
server_key: "2c617c3a74769298893d19901b4addc1"
server_key_iv: "58f9104a3b6d9f574661115932efdb23"
server_key_version: "encryptionkey@0"
I20260926 13:45:45.527750 22423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance:
uuid: "3e84e6cc58604da0b7a4236545da6df0"
format_stamp: "Formatted at 2026-09-26 13:45:45 on dist-test-slave-hc1k"
server_key: "2c617c3a74769298893d19901b4addc1"
server_key_iv: "58f9104a3b6d9f574661115932efdb23"
server_key_version: "encryptionkey@0"
I20260926 13:45:45.530963 22423 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 13:45:45.533187 22437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:45.534272 22423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:45.534359 22423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "3e84e6cc58604da0b7a4236545da6df0"
format_stamp: "Formatted at 2026-09-26 13:45:45 on dist-test-slave-hc1k"
server_key: "2c617c3a74769298893d19901b4addc1"
server_key_iv: "58f9104a3b6d9f574661115932efdb23"
server_key_version: "encryptionkey@0"
I20260926 13:45:45.534449 22423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:45.554746 22423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:45.555338 22423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:45.555491 22423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:45.556030 22423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:45.556869 22423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:45.556911 22423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:45.556948 22423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:45.556977 22423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:45.565246 22423 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:37151
I20260926 13:45:45.565285 22550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:37151 every 8 connection(s)
I20260926 13:45:45.566030 22423 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:45:45.574266 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 22423
I20260926 13:45:45.574373 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance
I20260926 13:45:45.574685 12969 external_mini_cluster.cc:1428] Setting key 064b56105e5cb8b2a31733ba3160f7eb
I20260926 13:45:45.575184 22271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:45.567727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:53071 (local address 127.12.170.126:42457)
0926 13:45:45.567847 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:45.567851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:45.568818 (+ 967us) server_negotiation.cc:407] Connection header received
0926 13:45:45.570233 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:45.570236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:45.570298 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:45.570392 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:45.571387 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:45.572129 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:45.573325 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:45.573534 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:45.573620 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:45.574713 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:45.574716 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:45.574718 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:45:45.574732 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:45.574755 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:45:45.574765 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:45.574843 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:45.574987 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 13:45:45.575045 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 13:45:45.576126 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:45.567993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:42457 (local address 127.12.170.66:53071)
0926 13:45:45.568678 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:45.568716 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:45.569959 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:45.570611 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:45.570620 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:45.570730 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:45.571244 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:45.571257 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:45.572237 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:45.572239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:45.573223 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:45.573232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:45.573695 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:45.573699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:45.573836 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:45.574391 (+ 555us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:45:45.574393 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:45.574444 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:45:45.574521 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:45.574844 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:45.574853 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:45.574859 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 13:45:45.574974 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 13:45:45.575251 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":107,"threads_started":1}
I20260926 13:45:45.576643 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:0
--local_ip_for_outbound_sockets=127.12.170.67
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:42457
--builtin_ntp_servers=127.12.170.84:46679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 13:45:45.577144 22551 heartbeater.cc:347] Connected to a master server at 127.12.170.126:42457
I20260926 13:45:45.577373 22551 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:45.577945 22551 heartbeater.cc:511] Master 127.12.170.126:42457 requested a full tablet report, sending...
I20260926 13:45:45.578922 22232 ts_manager.cc:194] Registered new tserver with Master: 3e84e6cc58604da0b7a4236545da6df0 (127.12.170.66:37151)
I20260926 13:45:45.579450 22232 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.170.66:53071
W20260926 13:45:45.680151 22554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:45.680368 22554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:45.680415 22554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:45.684332 22554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:45.684377 22554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:45.684445 22554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:45:45.688881 22554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:46679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:42457
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:45.689751 22554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:45.690960 22554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:45.699616 22560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:45.699757 22554 server_base.cc:1034] running on GCE node
W20260926 13:45:45.699688 22559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:45.699636 22562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:45.700178 22554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:45.700749 22554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:45.701894 22554 hybrid_clock.cc:648] HybridClock initialized: now 1790430345701875 us; error 30 us; skew 500 ppm
I20260926 13:45:45.703510 22554 webserver.cc:460] Webserver started at http://127.12.170.67:38115/ using document root <none> and password file <none>
I20260926 13:45:45.704069 22554 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:45.704134 22554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:45.704305 22554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:45.706276 22554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/data/instance:
uuid: "008bbb98ffdf471f979d040476f7be68"
format_stamp: "Formatted at 2026-09-26 13:45:45 on dist-test-slave-hc1k"
server_key: "9d29ba72ccbb037a2b759411fe5dd67f"
server_key_iv: "31cd7094d03f4d2a9ea5f76dee1228dd"
server_key_version: "encryptionkey@0"
I20260926 13:45:45.706781 22554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance:
uuid: "008bbb98ffdf471f979d040476f7be68"
format_stamp: "Formatted at 2026-09-26 13:45:45 on dist-test-slave-hc1k"
server_key: "9d29ba72ccbb037a2b759411fe5dd67f"
server_key_iv: "31cd7094d03f4d2a9ea5f76dee1228dd"
server_key_version: "encryptionkey@0"
I20260926 13:45:45.710076 22554 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20260926 13:45:45.712337 22568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:45.713439 22554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:45.713534 22554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "008bbb98ffdf471f979d040476f7be68"
format_stamp: "Formatted at 2026-09-26 13:45:45 on dist-test-slave-hc1k"
server_key: "9d29ba72ccbb037a2b759411fe5dd67f"
server_key_iv: "31cd7094d03f4d2a9ea5f76dee1228dd"
server_key_version: "encryptionkey@0"
I20260926 13:45:45.713615 22554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:45.732437 22554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:45.733004 22554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:45.733140 22554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:45.733596 22554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:45.734372 22554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:45.734416 22554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:45.734450 22554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:45.734469 22554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:45.742278 22554 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:45941
I20260926 13:45:45.742331 22681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:45941 every 8 connection(s)
I20260926 13:45:45.743036 22554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:45:45.750847 22271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:45.744558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:54669 (local address 127.12.170.126:42457)
0926 13:45:45.744711 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:45.744716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:45.745649 (+ 933us) server_negotiation.cc:407] Connection header received
0926 13:45:45.746853 (+ 1204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:45.746855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:45.746900 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:45.746959 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:45.747749 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:45.748235 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:45.749284 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:45.749481 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:45.749545 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:45.750398 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:45.750401 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:45.750403 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:45:45.750415 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:45.750434 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:45:45.750441 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:45.750495 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:45.750694 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 13:45:45.750742 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 13:45:45.751587 22684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:45.744915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:42457 (local address 127.12.170.67:54669)
0926 13:45:45.745500 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:45.745536 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:45.746681 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:45.747093 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:45.747099 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:45.747173 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:45.747614 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:45.747626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:45.748345 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:45.748348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:45.749199 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:45.749208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:45.749601 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:45.749604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:45.749728 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:45.750187 (+ 459us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:45:45.750189 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:45.750222 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:45:45.750277 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:45.750566 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:45.750574 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:45.750579 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:45:45.750694 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 13:45:45.750893 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":111,"threads_started":1}
I20260926 13:45:45.752425 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 22554
I20260926 13:45:45.752526 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance
I20260926 13:45:45.752761 22682 heartbeater.cc:347] Connected to a master server at 127.12.170.126:42457
I20260926 13:45:45.752808 12969 external_mini_cluster.cc:1428] Setting key b7039058e6912950015fbe3bd477fc55
I20260926 13:45:45.752959 22682 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:45.753449 22682 heartbeater.cc:511] Master 127.12.170.126:42457 requested a full tablet report, sending...
I20260926 13:45:45.754386 22231 ts_manager.cc:194] Registered new tserver with Master: 008bbb98ffdf471f979d040476f7be68 (127.12.170.67:45941)
I20260926 13:45:45.755057 22231 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.170.67:54669
I20260926 13:45:45.766214 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 13:45:45.768754 22271 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:45.767958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60046 (local address 127.12.170.126:42457)
0926 13:45:45.768084 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:45.768088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:45.768220 (+ 132us) server_negotiation.cc:407] Connection header received
0926 13:45:45.768332 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:45.768334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:45.768369 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:45.768428 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:45.768648 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60046: BlockingRecv error: recv got EOF from 127.0.0.1:60046 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 13:45:45.769029 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 22292
I20260926 13:45:45.773744 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 22423
I20260926 13:45:45.778990 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 22554
I20260926 13:45:45.783092 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 22201
2026-09-26T13:45:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (823 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T13:45:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:45:45Z Disabled control of system clock
I20260926 13:45:45.808353 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:34097
--webserver_interface=127.12.170.126
--webserver_port=0
--builtin_ntp_servers=127.12.170.84:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:34097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 13:45:45.912426 22693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:45.912724 22693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:45.912783 22693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:45.916718 22693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:45:45.916767 22693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:45.916790 22693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:45.916811 22693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:45:45.916839 22693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:45:45.921828 22693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:34097
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:34097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:45.922780 22693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:45.924067 22693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:45.931798 22698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:45.931852 22693 server_base.cc:1034] running on GCE node
W20260926 13:45:45.931988 22699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:45.931820 22701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:45.932561 22693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:45.933519 22693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:45.934666 22693 hybrid_clock.cc:648] HybridClock initialized: now 1790430345934652 us; error 31 us; skew 500 ppm
I20260926 13:45:45.936390 22693 webserver.cc:460] Webserver started at http://127.12.170.126:44445/ using document root <none> and password file <none>
I20260926 13:45:45.936882 22693 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:45.936942 22693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:45.937110 22693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:45.938974 22693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "7958b6b605b14f6abe947656bfa4d2a3"
format_stamp: "Formatted at 2026-09-26 13:45:45 on dist-test-slave-hc1k"
server_key: "fd07bd080f00a292a67dcbddd8f35dc7"
server_key_iv: "0d7374d30ff41cda6a1581018b43e67d"
server_key_version: "encryptionkey@0"
I20260926 13:45:45.939468 22693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "7958b6b605b14f6abe947656bfa4d2a3"
format_stamp: "Formatted at 2026-09-26 13:45:45 on dist-test-slave-hc1k"
server_key: "fd07bd080f00a292a67dcbddd8f35dc7"
server_key_iv: "0d7374d30ff41cda6a1581018b43e67d"
server_key_version: "encryptionkey@0"
I20260926 13:45:45.942752 22693 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:45:45.944852 22707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:45.945921 22693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:45.946022 22693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "7958b6b605b14f6abe947656bfa4d2a3"
format_stamp: "Formatted at 2026-09-26 13:45:45 on dist-test-slave-hc1k"
server_key: "fd07bd080f00a292a67dcbddd8f35dc7"
server_key_iv: "0d7374d30ff41cda6a1581018b43e67d"
server_key_version: "encryptionkey@0"
I20260926 13:45:45.946115 22693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:45.950181 22693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:45.950690 22693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:45.950829 22693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:45.956876 22693 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:34097
I20260926 13:45:45.956889 22759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:34097 every 8 connection(s)
I20260926 13:45:45.957684 22693 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:45:45.960582 22760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:45.962575 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 22693
I20260926 13:45:45.962674 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:45:45.962934 12969 external_mini_cluster.cc:1428] Setting key d72d9722252a88b88c57e1f7f2d977ed
I20260926 13:45:45.965996 22760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7958b6b605b14f6abe947656bfa4d2a3: Bootstrap starting.
I20260926 13:45:45.967901 22763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:45.964233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44440 (local address 127.12.170.126:34097)
0926 13:45:45.964791 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:45.964804 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:45.964836 (+ 32us) server_negotiation.cc:407] Connection header received
0926 13:45:45.965434 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:45.965450 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:45.965770 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:45.966051 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:45.966753 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:45.966758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:45.966768 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:45:45.966778 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:45.966828 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:45:45.966843 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:45.966955 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:45.967042 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 13:45:45.967264 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":108,"threads_started":1}
I20260926 13:45:45.968127 22760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7958b6b605b14f6abe947656bfa4d2a3: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:45.968804 22760 log.cc:824] T 00000000000000000000000000000000 P 7958b6b605b14f6abe947656bfa4d2a3: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:45.970569 22760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7958b6b605b14f6abe947656bfa4d2a3: No bootstrap required, opened a new log
I20260926 13:45:45.973680 22760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7958b6b605b14f6abe947656bfa4d2a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7958b6b605b14f6abe947656bfa4d2a3" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34097 } }
I20260926 13:45:45.973871 22760 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7958b6b605b14f6abe947656bfa4d2a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:45.973908 22760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7958b6b605b14f6abe947656bfa4d2a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7958b6b605b14f6abe947656bfa4d2a3, State: Initialized, Role: FOLLOWER
I20260926 13:45:45.974255 22760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7958b6b605b14f6abe947656bfa4d2a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7958b6b605b14f6abe947656bfa4d2a3" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34097 } }
I20260926 13:45:45.974364 22760 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7958b6b605b14f6abe947656bfa4d2a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:45:45.974406 22760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7958b6b605b14f6abe947656bfa4d2a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:45:45.974469 22760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7958b6b605b14f6abe947656bfa4d2a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:45.975220 22760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7958b6b605b14f6abe947656bfa4d2a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7958b6b605b14f6abe947656bfa4d2a3" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34097 } }
I20260926 13:45:45.975562 22760 leader_election.cc:302] T 00000000000000000000000000000000 P 7958b6b605b14f6abe947656bfa4d2a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7958b6b605b14f6abe947656bfa4d2a3; no voters:
I20260926 13:45:45.975879 22760 leader_election.cc:288] T 00000000000000000000000000000000 P 7958b6b605b14f6abe947656bfa4d2a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:45:45.975947 22765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7958b6b605b14f6abe947656bfa4d2a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:45.976161 22765 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7958b6b605b14f6abe947656bfa4d2a3 [term 1 LEADER]: Becoming Leader. State: Replica: 7958b6b605b14f6abe947656bfa4d2a3, State: Running, Role: LEADER
I20260926 13:45:45.976481 22765 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7958b6b605b14f6abe947656bfa4d2a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7958b6b605b14f6abe947656bfa4d2a3" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34097 } }
I20260926 13:45:45.976933 22760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7958b6b605b14f6abe947656bfa4d2a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:45:45.977864 22766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7958b6b605b14f6abe947656bfa4d2a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7958b6b605b14f6abe947656bfa4d2a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7958b6b605b14f6abe947656bfa4d2a3" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34097 } } }
I20260926 13:45:45.977980 22766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7958b6b605b14f6abe947656bfa4d2a3 [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:45.978339 22774 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:45:45.978298 22767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7958b6b605b14f6abe947656bfa4d2a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7958b6b605b14f6abe947656bfa4d2a3. Latest consensus state: current_term: 1 leader_uuid: "7958b6b605b14f6abe947656bfa4d2a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7958b6b605b14f6abe947656bfa4d2a3" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34097 } } }
I20260926 13:45:45.978447 22767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7958b6b605b14f6abe947656bfa4d2a3 [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:45.981421 22774 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:45:45.986644 22774 catalog_manager.cc:1347] Generated new cluster ID: 16570e207d944ac095019a92f8b197c8
I20260926 13:45:45.986708 22774 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:45:46.003288 22774 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:45:46.003612 22774 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:45:46.010119 22774 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7958b6b605b14f6abe947656bfa4d2a3: Generated new TSK 0
I20260926 13:45:46.010836 22774 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:45:46.013247 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:0
--local_ip_for_outbound_sockets=127.12.170.65
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:34097
--builtin_ntp_servers=127.12.170.84:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 13:45:46.116443 22784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:46.116675 22784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:46.116729 22784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:46.120648 22784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:46.120692 22784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:46.120762 22784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:45:46.125257 22784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:34097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:46.126161 22784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:46.127391 22784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:46.136188 22790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:46.136237 22789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:46.136209 22792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:46.136575 22784 server_base.cc:1034] running on GCE node
I20260926 13:45:46.136924 22784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:46.137499 22784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:46.138679 22784 hybrid_clock.cc:648] HybridClock initialized: now 1790430346138656 us; error 31 us; skew 500 ppm
I20260926 13:45:46.140336 22784 webserver.cc:460] Webserver started at http://127.12.170.65:46289/ using document root <none> and password file <none>
I20260926 13:45:46.140841 22784 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:46.140900 22784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:46.141073 22784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:46.142956 22784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/data/instance:
uuid: "6e2c174c10db43b9990747f222552740"
format_stamp: "Formatted at 2026-09-26 13:45:46 on dist-test-slave-hc1k"
server_key: "6e86d947e8720386fdb79958e5f8857e"
server_key_iv: "61dc2e3d44a0cd0eb15252b29c3a3692"
server_key_version: "encryptionkey@0"
I20260926 13:45:46.143451 22784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance:
uuid: "6e2c174c10db43b9990747f222552740"
format_stamp: "Formatted at 2026-09-26 13:45:46 on dist-test-slave-hc1k"
server_key: "6e86d947e8720386fdb79958e5f8857e"
server_key_iv: "61dc2e3d44a0cd0eb15252b29c3a3692"
server_key_version: "encryptionkey@0"
I20260926 13:45:46.146756 22784 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:45:46.149118 22798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:46.150218 22784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:46.150314 22784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "6e2c174c10db43b9990747f222552740"
format_stamp: "Formatted at 2026-09-26 13:45:46 on dist-test-slave-hc1k"
server_key: "6e86d947e8720386fdb79958e5f8857e"
server_key_iv: "61dc2e3d44a0cd0eb15252b29c3a3692"
server_key_version: "encryptionkey@0"
I20260926 13:45:46.150401 22784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:46.154528 22784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:46.155056 22784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:46.155196 22784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:46.155691 22784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:46.156483 22784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:46.156528 22784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:46.156584 22784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:46.156607 22784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:46.164353 22784 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:45727
I20260926 13:45:46.164394 22911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:45727 every 8 connection(s)
I20260926 13:45:46.165127 22784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:45:46.168726 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 22784
I20260926 13:45:46.168823 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance
I20260926 13:45:46.169078 12969 external_mini_cluster.cc:1428] Setting key 44acf36dc25829acd79db372cfd2af54
I20260926 13:45:46.170938 22763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:46.166727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:45251 (local address 127.12.170.126:34097)
0926 13:45:46.166890 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:46.166894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:46.167919 (+ 1025us) server_negotiation.cc:407] Connection header received
0926 13:45:46.169238 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:46.169242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:46.169285 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:46.169379 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:46.170481 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:46.170484 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:46.170487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:45:46.170498 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:46.170521 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:45:46.170531 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:46.170611 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:46.170777 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 13:45:46.170815 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 13:45:46.171128 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:0
--local_ip_for_outbound_sockets=127.12.170.66
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:34097
--builtin_ntp_servers=127.12.170.84:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 13:45:46.171818 22914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:46.167101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34097 (local address 127.12.170.65:45251)
0926 13:45:46.167782 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:46.167822 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:46.169015 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:46.169579 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:46.169588 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:46.169690 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:46.170212 (+ 522us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:45:46.170213 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:46.170250 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:45:46.170331 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:46.170633 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:46.170641 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:46.170646 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:45:46.170797 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 13:45:46.170971 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":105,"threads_started":1}
I20260926 13:45:46.173120 22912 heartbeater.cc:347] Connected to a master server at 127.12.170.126:34097
I20260926 13:45:46.173342 22912 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:46.173719 22912 heartbeater.cc:511] Master 127.12.170.126:34097 requested a full tablet report, sending...
I20260926 13:45:46.175297 22724 ts_manager.cc:194] Registered new tserver with Master: 6e2c174c10db43b9990747f222552740 (127.12.170.65:45727)
W20260926 13:45:46.279569 22915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:46.279819 22915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:46.279855 22915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:46.283778 22915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:46.283823 22915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:46.283874 22915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:45:46.288383 22915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:34097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:46.289247 22915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:46.290446 22915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:46.299197 22920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:46.299233 22915 server_base.cc:1034] running on GCE node
W20260926 13:45:46.299237 22921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:46.299429 22923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:46.299808 22915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:46.300355 22915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:46.301502 22915 hybrid_clock.cc:648] HybridClock initialized: now 1790430346301477 us; error 37 us; skew 500 ppm
I20260926 13:45:46.303206 22915 webserver.cc:460] Webserver started at http://127.12.170.66:46863/ using document root <none> and password file <none>
I20260926 13:45:46.303751 22915 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:46.303818 22915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:46.304000 22915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:46.305858 22915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/data/instance:
uuid: "6774df2b1f8342408dd1522fe8297f4d"
format_stamp: "Formatted at 2026-09-26 13:45:46 on dist-test-slave-hc1k"
server_key: "48dad48bcfc68b8721970e0c14d96e78"
server_key_iv: "a2930879810ab87ee08d4fc7d3572346"
server_key_version: "encryptionkey@0"
I20260926 13:45:46.306350 22915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance:
uuid: "6774df2b1f8342408dd1522fe8297f4d"
format_stamp: "Formatted at 2026-09-26 13:45:46 on dist-test-slave-hc1k"
server_key: "48dad48bcfc68b8721970e0c14d96e78"
server_key_iv: "a2930879810ab87ee08d4fc7d3572346"
server_key_version: "encryptionkey@0"
I20260926 13:45:46.309598 22915 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:45:46.311839 22929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:46.312981 22915 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:45:46.313081 22915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "6774df2b1f8342408dd1522fe8297f4d"
format_stamp: "Formatted at 2026-09-26 13:45:46 on dist-test-slave-hc1k"
server_key: "48dad48bcfc68b8721970e0c14d96e78"
server_key_iv: "a2930879810ab87ee08d4fc7d3572346"
server_key_version: "encryptionkey@0"
I20260926 13:45:46.313174 22915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:46.317317 22915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:46.317835 22915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:46.317970 22915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:46.318425 22915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:46.319211 22915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:46.319253 22915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:46.319291 22915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:46.319319 22915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:46.327157 22915 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:43307
I20260926 13:45:46.327180 23042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:43307 every 8 connection(s)
I20260926 13:45:46.327936 22915 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:45:46.333261 22763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:46.329519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:36523 (local address 127.12.170.126:34097)
0926 13:45:46.329636 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:46.329640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:46.330572 (+ 932us) server_negotiation.cc:407] Connection header received
0926 13:45:46.331789 (+ 1217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:46.331792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:46.331829 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:46.331897 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:46.332850 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:46.332854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:46.332857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:45:46.332869 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:46.332889 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:45:46.332897 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:46.332970 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:46.333092 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 13:45:46.333133 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:45:46.333978 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:46.329857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34097 (local address 127.12.170.66:36523)
0926 13:45:46.330438 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:46.330474 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:46.331563 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:46.332027 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:46.332033 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:46.332113 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:46.332610 (+ 497us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:45:46.332612 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:46.332646 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:45:46.332708 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:46.332962 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:46.332969 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:46.332974 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:45:46.333116 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 13:45:46.333330 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":105,"threads_started":1}
I20260926 13:45:46.334919 23043 heartbeater.cc:347] Connected to a master server at 127.12.170.126:34097
I20260926 13:45:46.335100 23043 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:46.335428 23043 heartbeater.cc:511] Master 127.12.170.126:34097 requested a full tablet report, sending...
I20260926 13:45:46.336346 22724 ts_manager.cc:194] Registered new tserver with Master: 6774df2b1f8342408dd1522fe8297f4d (127.12.170.66:43307)
I20260926 13:45:46.336671 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 22915
I20260926 13:45:46.336788 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance
I20260926 13:45:46.337081 12969 external_mini_cluster.cc:1428] Setting key 62f0fea1e5eca1ad0bbd24263ef34452
I20260926 13:45:46.338670 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:0
--local_ip_for_outbound_sockets=127.12.170.67
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:34097
--builtin_ntp_servers=127.12.170.84:36047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 13:45:46.444123 23046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:46.444335 23046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:46.444372 23046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:46.448263 23046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:46.448310 23046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:46.448361 23046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:45:46.452976 23046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:36047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:34097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:46.453907 23046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:46.455186 23046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:46.463882 23052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:46.463912 23054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:46.463930 23046 server_base.cc:1034] running on GCE node
W20260926 13:45:46.463992 23051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:46.464454 23046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:46.465085 23046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:46.466241 23046 hybrid_clock.cc:648] HybridClock initialized: now 1790430346466225 us; error 29 us; skew 500 ppm
I20260926 13:45:46.467983 23046 webserver.cc:460] Webserver started at http://127.12.170.67:34299/ using document root <none> and password file <none>
I20260926 13:45:46.468480 23046 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:46.468538 23046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:46.468715 23046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:46.470559 23046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/data/instance:
uuid: "40d43015e46c45059400bbe52c6a9f14"
format_stamp: "Formatted at 2026-09-26 13:45:46 on dist-test-slave-hc1k"
server_key: "7851cfc6ee4df43635da4d8ce5d0c50a"
server_key_iv: "fa34cdf3b199359b331eaf4d72895db4"
server_key_version: "encryptionkey@0"
I20260926 13:45:46.471048 23046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance:
uuid: "40d43015e46c45059400bbe52c6a9f14"
format_stamp: "Formatted at 2026-09-26 13:45:46 on dist-test-slave-hc1k"
server_key: "7851cfc6ee4df43635da4d8ce5d0c50a"
server_key_iv: "fa34cdf3b199359b331eaf4d72895db4"
server_key_version: "encryptionkey@0"
I20260926 13:45:46.474347 23046 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 13:45:46.476632 23060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:46.477788 23046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:45:46.477885 23046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "40d43015e46c45059400bbe52c6a9f14"
format_stamp: "Formatted at 2026-09-26 13:45:46 on dist-test-slave-hc1k"
server_key: "7851cfc6ee4df43635da4d8ce5d0c50a"
server_key_iv: "fa34cdf3b199359b331eaf4d72895db4"
server_key_version: "encryptionkey@0"
I20260926 13:45:46.477979 23046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:46.482146 23046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:46.482678 23046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:46.482831 23046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:46.483309 23046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:46.484148 23046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:46.484201 23046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:46.484249 23046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:46.484277 23046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:46.491956 23046 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:35971
I20260926 13:45:46.491979 23173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:35971 every 8 connection(s)
I20260926 13:45:46.492709 23046 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:45:46.493577 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 23046
I20260926 13:45:46.493655 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance
I20260926 13:45:46.493902 12969 external_mini_cluster.cc:1428] Setting key 527be5ecc467de1c1ff067a6cffaef20
I20260926 13:45:46.498224 22763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:46.494348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:53923 (local address 127.12.170.126:34097)
0926 13:45:46.494472 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:46.494475 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:46.495694 (+ 1219us) server_negotiation.cc:407] Connection header received
0926 13:45:46.496945 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:46.496947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:46.496981 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:46.497043 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:46.497937 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:46.497939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:46.497940 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:45:46.497947 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:46.497960 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:45:46.497966 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:46.498018 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:46.498101 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 13:45:46.498132 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 13:45:46.499009 23176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:46.494797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34097 (local address 127.12.170.67:53923)
0926 13:45:46.495534 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:46.495583 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:46.496776 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:46.497161 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:46.497167 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:46.497245 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:46.497741 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:45:46.497743 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:46.497778 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:45:46.497839 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:46.498010 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:46.498014 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:46.498017 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:45:46.498125 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 13:45:46.498301 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":130,"threads_started":1}
I20260926 13:45:46.499930 23174 heartbeater.cc:347] Connected to a master server at 127.12.170.126:34097
I20260926 13:45:46.500140 23174 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:46.500463 23174 heartbeater.cc:511] Master 127.12.170.126:34097 requested a full tablet report, sending...
I20260926 13:45:46.501392 22724 ts_manager.cc:194] Registered new tserver with Master: 40d43015e46c45059400bbe52c6a9f14 (127.12.170.67:35971)
I20260926 13:45:46.507325 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 13:45:46.509784 22763 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:45:46.509084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44444 (local address 127.12.170.126:34097)
0926 13:45:46.509208 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:46.509212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:46.509322 (+ 110us) server_negotiation.cc:407] Connection header received
0926 13:45:46.509412 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:46.509414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:46.509449 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:46.509508 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:46.509699 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44444: BlockingRecv error: recv got EOF from 127.0.0.1:44444 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 13:45:46.510267 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 22784
I20260926 13:45:46.515249 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 22915
I20260926 13:45:46.519944 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 23046
I20260926 13:45:46.524255 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 22693
2026-09-26T13:45:46Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (740 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:45:46 dist-test-slave-hc1k krb5kdc[23184](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:45:46 dist-test-slave-hc1k krb5kdc[23184](info): set up 2 sockets
Sep 26 13:45:46 dist-test-slave-hc1k krb5kdc[23184](info): commencing operation
krb5kdc: starting...
W20260926 13:45:47.464354 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:45:47 dist-test-slave-hc1k krb5kdc[23184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:45:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:45:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:45:47.574010 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:44561
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:42455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:44561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:47.675397 23197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:47.675637 23197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:47.675719 23197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:47.679567 23197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:45:47.679612 23197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:47.679630 23197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:47.679644 23197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:45:47.679679 23197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:45:47.684643 23197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:42455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:44561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:44561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:47.685459 23197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:47.686329 23197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:47.693948 23205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:47.693982 23202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:47.694088 23197 server_base.cc:1034] running on GCE node
W20260926 13:45:47.693988 23203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:47.694679 23197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:47.695673 23197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:47.696856 23197 hybrid_clock.cc:648] HybridClock initialized: now 1790430347696833 us; error 39 us; skew 500 ppm
Sep 26 13:45:47 dist-test-slave-hc1k krb5kdc[23184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430347, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:47.699424 23197 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:45:47.700277 23197 webserver.cc:460] Webserver started at http://127.12.170.126:34033/ using document root <none> and password file <none>
I20260926 13:45:47.700765 23197 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:47.700831 23197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:47.701010 23197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:47.702999 23197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "21090777a1c54e3190a322aadec69429"
format_stamp: "Formatted at 2026-09-26 13:45:47 on dist-test-slave-hc1k"
server_key: "81cd87cf46f0dd690bd0ca587371375a"
server_key_iv: "63abe098dcf33de87d8ad0ca6a0abf26"
server_key_version: "encryptionkey@0"
I20260926 13:45:47.703533 23197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "21090777a1c54e3190a322aadec69429"
format_stamp: "Formatted at 2026-09-26 13:45:47 on dist-test-slave-hc1k"
server_key: "81cd87cf46f0dd690bd0ca587371375a"
server_key_iv: "63abe098dcf33de87d8ad0ca6a0abf26"
server_key_version: "encryptionkey@0"
I20260926 13:45:47.707139 23197 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:45:47.709416 23212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:47.710539 23197 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:47.710652 23197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "21090777a1c54e3190a322aadec69429"
format_stamp: "Formatted at 2026-09-26 13:45:47 on dist-test-slave-hc1k"
server_key: "81cd87cf46f0dd690bd0ca587371375a"
server_key_iv: "63abe098dcf33de87d8ad0ca6a0abf26"
server_key_version: "encryptionkey@0"
I20260926 13:45:47.710743 23197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:47.725899 23197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:47.728868 23197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:47.729048 23197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:47.735621 23197 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:44561
I20260926 13:45:47.735888 23264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:44561 every 8 connection(s)
I20260926 13:45:47.736508 23197 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:45:47.738502 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 23197
I20260926 13:45:47.738611 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:45:47.738885 12969 external_mini_cluster.cc:1428] Setting key abe7ade56cdaf74321fae072595b1d70
I20260926 13:45:47.739214 23265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:47.745285 23265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21090777a1c54e3190a322aadec69429: Bootstrap starting.
Sep 26 13:45:47 dist-test-slave-hc1k krb5kdc[23184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:47.747714 23265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21090777a1c54e3190a322aadec69429: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:47.748375 23265 log.cc:824] T 00000000000000000000000000000000 P 21090777a1c54e3190a322aadec69429: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:47.750042 23265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21090777a1c54e3190a322aadec69429: No bootstrap required, opened a new log
I20260926 13:45:47.752074 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:47.740113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44682 (local address 127.12.170.126:44561)
0926 13:45:47.740484 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:47.740498 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:47.740531 (+ 33us) server_negotiation.cc:407] Connection header received
0926 13:45:47.741124 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:47.741138 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:47.741443 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:47.741699 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:47.742527 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:47.743682 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:47.744771 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:47.745221 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:47.745346 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:47.747907 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:47.747912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:47.747923 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:47.747948 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:47.749591 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:47.750012 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:47.750016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:47.750020 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:47.750157 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:47.750431 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:47.750434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:47.750435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:47.750767 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:47.750898 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:47.751164 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 13:45:47.751398 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":102,"threads_started":1}
I20260926 13:45:47.753651 23265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21090777a1c54e3190a322aadec69429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21090777a1c54e3190a322aadec69429" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 44561 } }
I20260926 13:45:47.753880 23265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21090777a1c54e3190a322aadec69429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:47.753937 23265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21090777a1c54e3190a322aadec69429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21090777a1c54e3190a322aadec69429, State: Initialized, Role: FOLLOWER
I20260926 13:45:47.754302 23265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21090777a1c54e3190a322aadec69429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21090777a1c54e3190a322aadec69429" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 44561 } }
I20260926 13:45:47.754412 23265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21090777a1c54e3190a322aadec69429 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:45:47.754454 23265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21090777a1c54e3190a322aadec69429 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:45:47.754518 23265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21090777a1c54e3190a322aadec69429 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:47.755204 23265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21090777a1c54e3190a322aadec69429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21090777a1c54e3190a322aadec69429" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 44561 } }
I20260926 13:45:47.755462 23265 leader_election.cc:302] T 00000000000000000000000000000000 P 21090777a1c54e3190a322aadec69429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21090777a1c54e3190a322aadec69429; no voters:
I20260926 13:45:47.755703 23265 leader_election.cc:288] T 00000000000000000000000000000000 P 21090777a1c54e3190a322aadec69429 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:45:47.755795 23270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21090777a1c54e3190a322aadec69429 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:47.756023 23270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21090777a1c54e3190a322aadec69429 [term 1 LEADER]: Becoming Leader. State: Replica: 21090777a1c54e3190a322aadec69429, State: Running, Role: LEADER
I20260926 13:45:47.756292 23270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21090777a1c54e3190a322aadec69429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21090777a1c54e3190a322aadec69429" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 44561 } }
I20260926 13:45:47.756718 23265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21090777a1c54e3190a322aadec69429 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:45:47.757617 23272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21090777a1c54e3190a322aadec69429 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21090777a1c54e3190a322aadec69429. Latest consensus state: current_term: 1 leader_uuid: "21090777a1c54e3190a322aadec69429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21090777a1c54e3190a322aadec69429" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 44561 } } }
I20260926 13:45:47.757743 23272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21090777a1c54e3190a322aadec69429 [sys.catalog]: This master's current role is: LEADER
I20260926 13:45:47.758008 23271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21090777a1c54e3190a322aadec69429 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21090777a1c54e3190a322aadec69429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21090777a1c54e3190a322aadec69429" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 44561 } } }
I20260926 13:45:47.758104 23271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21090777a1c54e3190a322aadec69429 [sys.catalog]: This master's current role is: LEADER
W20260926 13:45:47.761200 23285 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 21090777a1c54e3190a322aadec69429: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:45:47.761258 23285 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:45:47.761322 23279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:45:47.761969 23279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:45:47.766535 23279 catalog_manager.cc:1347] Generated new cluster ID: 04f1e9edbfaa40ea92f909d6d5233a9f
I20260926 13:45:47.766602 23279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:45:47.782336 23279 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:45:47.783528 23279 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:45:47.795204 23279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21090777a1c54e3190a322aadec69429: Generated new TSK 0
I20260926 13:45:47.795969 23279 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.170.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
I20260926 13:45:47.822536 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:0
--local_ip_for_outbound_sockets=127.12.170.65
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:44561
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:42455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:47.927037 23291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:47.927259 23291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:47.927312 23291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:47.931191 23291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:47.931245 23291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:47.931321 23291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:45:47.936270 23291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:42455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:44561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:47.937146 23291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:47.938028 23291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:47.946794 23296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:47.946973 23299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:47.946836 23291 server_base.cc:1034] running on GCE node
W20260926 13:45:47.946797 23297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:47.947764 23291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:47.948359 23291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:47.949488 23291 hybrid_clock.cc:648] HybridClock initialized: now 1790430347949461 us; error 43 us; skew 500 ppm
Sep 26 13:45:47 dist-test-slave-hc1k krb5kdc[23184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430347, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:47.952517 23291 init.cc:377] Logged in from keytab as kudu/127.12.170.65@KRBTEST.COM (short username kudu)
I20260926 13:45:47.953299 23291 webserver.cc:460] Webserver started at http://127.12.170.65:37079/ using document root <none> and password file <none>
I20260926 13:45:47.953786 23291 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:47.953846 23291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:47.954015 23291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:47.955902 23291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/data/instance:
uuid: "f258fdc494224a638e10faf28d7014db"
format_stamp: "Formatted at 2026-09-26 13:45:47 on dist-test-slave-hc1k"
server_key: "e42dbdc2b0528feab791b0ad2f533363"
server_key_iv: "88052e84cee43c6a3d598d76b782b166"
server_key_version: "encryptionkey@0"
I20260926 13:45:47.956476 23291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance:
uuid: "f258fdc494224a638e10faf28d7014db"
format_stamp: "Formatted at 2026-09-26 13:45:47 on dist-test-slave-hc1k"
server_key: "e42dbdc2b0528feab791b0ad2f533363"
server_key_iv: "88052e84cee43c6a3d598d76b782b166"
server_key_version: "encryptionkey@0"
I20260926 13:45:47.959985 23291 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 13:45:47.962095 23306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:47.963343 23291 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:45:47.963469 23291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "f258fdc494224a638e10faf28d7014db"
format_stamp: "Formatted at 2026-09-26 13:45:47 on dist-test-slave-hc1k"
server_key: "e42dbdc2b0528feab791b0ad2f533363"
server_key_iv: "88052e84cee43c6a3d598d76b782b166"
server_key_version: "encryptionkey@0"
I20260926 13:45:47.963572 23291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:47.981137 23291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:47.983798 23291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:47.983964 23291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:47.984477 23291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:47.985209 23291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:47.985252 23291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:47.985291 23291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:47.985319 23291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:47.993192 23291 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:38093
I20260926 13:45:47.993238 23419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:38093 every 8 connection(s)
I20260926 13:45:47.994033 23291 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:45:47.997721 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 23291
I20260926 13:45:47.997824 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance
I20260926 13:45:47.998088 12969 external_mini_cluster.cc:1428] Setting key ce0797e89a78a5c09dbb9a8705791949
Sep 26 13:45:47 dist-test-slave-hc1k krb5kdc[23184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430347, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.170.66@KRBTEST.COM; defaulting to no policy
I20260926 13:45:48.007563 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:47.995609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:54319 (local address 127.12.170.126:44561)
0926 13:45:47.995773 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:47.995777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:47.996541 (+ 764us) server_negotiation.cc:407] Connection header received
0926 13:45:47.997891 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:47.997895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:47.997949 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:47.998035 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:47.999283 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:47.999959 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.001426 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.001736 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.001842 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:48.004344 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:48.004347 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:48.004349 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:48.004372 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:48.005742 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:48.006206 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:48.006210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:48.006211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:48.006287 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:48.006574 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:48.006578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:48.006580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:48.006759 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:48.006842 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:48.007283 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 13:45:48.007406 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 13:45:48.008334 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:47.995988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:44561 (local address 127.12.170.65:54319)
0926 13:45:47.996390 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:47.996428 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:47.997645 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:47.998222 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:47.998232 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:47.998637 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:47.999137 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:47.999150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.000093 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.000097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:48.001313 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:48.001323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.001858 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.001860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:48.001987 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:48.002493 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:48.002509 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:48.004181 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:48.005859 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:48.005864 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:48.005877 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:48.006115 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:48.006363 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:48.006366 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:48.006368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:48.006491 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:48.006853 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:48.006859 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:48.007073 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 13:45:48.007225 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 13:45:48.007516 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":106,"threads_started":1}
I20260926 13:45:48.009496 23420 heartbeater.cc:347] Connected to a master server at 127.12.170.126:44561
I20260926 13:45:48.009792 23420 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.66@KRBTEST.COM" created.
I20260926 13:45:48.010255 23420 heartbeater.cc:511] Master 127.12.170.126:44561 requested a full tablet report, sending...
I20260926 13:45:48.011678 23229 ts_manager.cc:194] Registered new tserver with Master: f258fdc494224a638e10faf28d7014db (127.12.170.65:38093)
I20260926 13:45:48.012739 23229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.65@KRBTEST.COM'} at 127.12.170.65:54319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
I20260926 13:45:48.023893 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:0
--local_ip_for_outbound_sockets=127.12.170.66
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:44561
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:42455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:48.128993 23425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:48.129298 23425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:48.129371 23425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:48.133421 23425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:48.133471 23425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:48.133525 23425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:45:48.138420 23425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:42455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:44561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:48.139264 23425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:48.140182 23425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:48.148721 23430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:48.148761 23425 server_base.cc:1034] running on GCE node
W20260926 13:45:48.148762 23431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:48.148936 23433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:48.149284 23425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:48.149822 23425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:48.150974 23425 hybrid_clock.cc:648] HybridClock initialized: now 1790430348150945 us; error 39 us; skew 500 ppm
Sep 26 13:45:48 dist-test-slave-hc1k krb5kdc[23184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430348, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:48.153429 23425 init.cc:377] Logged in from keytab as kudu/127.12.170.66@KRBTEST.COM (short username kudu)
I20260926 13:45:48.154187 23425 webserver.cc:460] Webserver started at http://127.12.170.66:45121/ using document root <none> and password file <none>
I20260926 13:45:48.154659 23425 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:48.154718 23425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:48.154878 23425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:48.156737 23425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/data/instance:
uuid: "ed0bb488dbde4044858fb7ed2c381ed8"
format_stamp: "Formatted at 2026-09-26 13:45:48 on dist-test-slave-hc1k"
server_key: "ad609a315aa3d7a5e4ea8ac5400f2040"
server_key_iv: "b2922b80e042fb51f3c4dd49d233fa2c"
server_key_version: "encryptionkey@0"
I20260926 13:45:48.157296 23425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance:
uuid: "ed0bb488dbde4044858fb7ed2c381ed8"
format_stamp: "Formatted at 2026-09-26 13:45:48 on dist-test-slave-hc1k"
server_key: "ad609a315aa3d7a5e4ea8ac5400f2040"
server_key_iv: "b2922b80e042fb51f3c4dd49d233fa2c"
server_key_version: "encryptionkey@0"
I20260926 13:45:48.160704 23425 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:45:48.162855 23440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:48.163939 23425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:45:48.164036 23425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "ed0bb488dbde4044858fb7ed2c381ed8"
format_stamp: "Formatted at 2026-09-26 13:45:48 on dist-test-slave-hc1k"
server_key: "ad609a315aa3d7a5e4ea8ac5400f2040"
server_key_iv: "b2922b80e042fb51f3c4dd49d233fa2c"
server_key_version: "encryptionkey@0"
I20260926 13:45:48.164122 23425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:48.186689 23425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:48.189474 23425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:48.189631 23425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:48.190163 23425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:48.190889 23425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:48.190932 23425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:48.190980 23425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:48.191009 23425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:48.200755 23425 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:40117
I20260926 13:45:48.200799 23553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:40117 every 8 connection(s)
I20260926 13:45:48.201702 23425 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:45:48.209307 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 23425
I20260926 13:45:48.209407 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance
I20260926 13:45:48.209735 12969 external_mini_cluster.cc:1428] Setting key 874ab01b7089fd8fcec0a0ef6a250a6a
Sep 26 13:45:48 dist-test-slave-hc1k krb5kdc[23184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430348, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:48.214781 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:48.203508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:46181 (local address 127.12.170.126:44561)
0926 13:45:48.203680 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:48.203685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:48.204468 (+ 783us) server_negotiation.cc:407] Connection header received
0926 13:45:48.205720 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:48.205723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:48.205768 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:48.205839 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:48.206872 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.207383 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.208477 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.208702 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.208783 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:48.211363 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:48.211368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:48.211371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:48.211399 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:48.212835 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:48.213376 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:48.213379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:48.213380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:48.213448 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:48.213767 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:48.213768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:48.213770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:48.213944 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:48.214032 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:48.214480 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 13:45:48.214630 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 13:45:48.215512 23556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:48.203919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:44561 (local address 127.12.170.66:46181)
0926 13:45:48.204330 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:48.204370 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:48.205534 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:48.205975 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:48.205982 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:48.206300 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:48.206767 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:48.206780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.207465 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.207467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:48.208371 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:48.208380 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.208831 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.208834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:48.208984 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:48.209551 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:48.209571 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:48.211208 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:48.212939 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:48.212944 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:48.212958 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:48.213293 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:48.213519 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:48.213522 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:48.213524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:48.213689 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:48.214046 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:48.214051 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:48.214263 (+ 212us) client_negotiation.cc:769] Sending connection context
0926 13:45:48.214420 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 13:45:48.214703 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":117,"threads_started":1}
I20260926 13:45:48.216532 23554 heartbeater.cc:347] Connected to a master server at 127.12.170.126:44561
I20260926 13:45:48.216751 23554 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:48.217167 23554 heartbeater.cc:511] Master 127.12.170.126:44561 requested a full tablet report, sending...
I20260926 13:45:48.218089 23229 ts_manager.cc:194] Registered new tserver with Master: ed0bb488dbde4044858fb7ed2c381ed8 (127.12.170.66:40117)
WARNING: no policy specified for kudu/127.12.170.67@KRBTEST.COM; defaulting to no policy
I20260926 13:45:48.218828 23229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.66@KRBTEST.COM'} at 127.12.170.66:46181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
I20260926 13:45:48.235291 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:0
--local_ip_for_outbound_sockets=127.12.170.67
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:44561
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:42455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:45:48.338649 23559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:45:48.338886 23559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:45:48.338927 23559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:45:48.342810 23559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:45:48.342856 23559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:45:48.342916 23559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:45:48.347895 23559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:42455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:44561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:45:48.348727 23559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:45:48.349599 23559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:45:48.358381 23567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:48.358501 23564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:45:48.358394 23565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:45:48.359005 23559 server_base.cc:1034] running on GCE node
I20260926 13:45:48.359355 23559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:45:48.359956 23559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:45:48.361106 23559 hybrid_clock.cc:648] HybridClock initialized: now 1790430348361094 us; error 25 us; skew 500 ppm
Sep 26 13:45:48 dist-test-slave-hc1k krb5kdc[23184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430348, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:45:48.363556 23559 init.cc:377] Logged in from keytab as kudu/127.12.170.67@KRBTEST.COM (short username kudu)
I20260926 13:45:48.364305 23559 webserver.cc:460] Webserver started at http://127.12.170.67:41597/ using document root <none> and password file <none>
I20260926 13:45:48.364766 23559 fs_manager.cc:359] Metadata directory not provided
I20260926 13:45:48.364816 23559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:45:48.364957 23559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:45:48.366804 23559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/data/instance:
uuid: "dd08b9c17a354084abff77df7048c0e0"
format_stamp: "Formatted at 2026-09-26 13:45:48 on dist-test-slave-hc1k"
server_key: "f38c51d593bd60cf012a74918583a5f0"
server_key_iv: "b2cb7834b7f4fe35dc28e6a2ebd5ebde"
server_key_version: "encryptionkey@0"
I20260926 13:45:48.367264 23559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance:
uuid: "dd08b9c17a354084abff77df7048c0e0"
format_stamp: "Formatted at 2026-09-26 13:45:48 on dist-test-slave-hc1k"
server_key: "f38c51d593bd60cf012a74918583a5f0"
server_key_iv: "b2cb7834b7f4fe35dc28e6a2ebd5ebde"
server_key_version: "encryptionkey@0"
I20260926 13:45:48.370626 23559 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 13:45:48.372716 23574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:48.373759 23559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:45:48.373844 23559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "dd08b9c17a354084abff77df7048c0e0"
format_stamp: "Formatted at 2026-09-26 13:45:48 on dist-test-slave-hc1k"
server_key: "f38c51d593bd60cf012a74918583a5f0"
server_key_iv: "b2cb7834b7f4fe35dc28e6a2ebd5ebde"
server_key_version: "encryptionkey@0"
I20260926 13:45:48.373934 23559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:45:48.398566 23559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:45:48.401543 23559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:45:48.401712 23559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:45:48.402238 23559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:45:48.403007 23559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:45:48.403049 23559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:48.403091 23559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:45:48.403118 23559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:45:48.411068 23559 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:41831
I20260926 13:45:48.411121 23687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:41831 every 8 connection(s)
I20260926 13:45:48.411931 23559 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:45:48.420375 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 23559
I20260926 13:45:48.420486 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance
Sep 26 13:45:48 dist-test-slave-hc1k krb5kdc[23184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430348, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:45:48.420791 12969 external_mini_cluster.cc:1428] Setting key d9a67bffb9974ae52b005ebbafa98fda
I20260926 13:45:48.424429 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:48.413577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:56519 (local address 127.12.170.126:44561)
0926 13:45:48.413714 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:48.413719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:48.414536 (+ 817us) server_negotiation.cc:407] Connection header received
0926 13:45:48.415982 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:48.415986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:48.416031 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:48.416097 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:48.417161 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.417666 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.418707 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.418907 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.418976 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:48.421285 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:48.421289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:48.421291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:48.421319 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:48.422733 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:48.423201 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:48.423203 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:48.423205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:48.423270 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:48.423537 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:48.423539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:48.423540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:48.423750 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:48.423820 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:48.424240 (+ 420us) server_negotiation.cc:299] Negotiation successful
0926 13:45:48.424321 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 13:45:48.425175 23690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:48.413987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:44561 (local address 127.12.170.67:56519)
0926 13:45:48.414403 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:48.414442 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:48.415782 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:48.416243 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:48.416249 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:48.416604 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:45:48.417056 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:48.417069 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.417751 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.417753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:48.418622 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:48.418631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.419006 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.419009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:48.419149 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:48.419615 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:45:48.419630 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:45:48.421136 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:45:48.422840 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:48.422845 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:48.422855 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:48.423117 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:48.423338 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:45:48.423340 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:45:48.423341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:45:48.423466 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:45:48.423829 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:45:48.423834 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:45:48.424057 (+ 223us) client_negotiation.cc:769] Sending connection context
0926 13:45:48.424215 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 13:45:48.424436 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":116,"threads_started":1}
I20260926 13:45:48.426165 23688 heartbeater.cc:347] Connected to a master server at 127.12.170.126:44561
I20260926 13:45:48.426357 23688 heartbeater.cc:464] Registering TS with master...
I20260926 13:45:48.426807 23688 heartbeater.cc:511] Master 127.12.170.126:44561 requested a full tablet report, sending...
I20260926 13:45:48.427759 23229 ts_manager.cc:194] Registered new tserver with Master: dd08b9c17a354084abff77df7048c0e0 (127.12.170.67:41831)
I20260926 13:45:48.428282 23229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.67@KRBTEST.COM'} at 127.12.170.67:56519
I20260926 13:45:48.434120 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:45:48.442225 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:48.435756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44684 (local address 127.12.170.126:44561)
0926 13:45:48.435851 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:48.435855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:48.436039 (+ 184us) server_negotiation.cc:407] Connection header received
0926 13:45:48.436154 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:48.436157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:48.436196 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:48.436255 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:45:48.436868 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.437354 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.438136 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.438350 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.438429 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:48.439550 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:45:48.439553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:45:48.439556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:45:48.439578 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:45:48.440704 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:48.441077 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:48.441104 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:48.441106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:48.441183 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:45:48.441439 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:45:48.441442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:45:48.441444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:45:48.441708 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:45:48.441827 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:48.442033 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 13:45:48.442124 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 13:45:48.444428 23229 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:45:48.447165 23229 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:45:48.459631 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:48.454426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.12.170.67:41831)
0926 13:45:48.454988 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:48.455000 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:48.455022 (+ 22us) server_negotiation.cc:407] Connection header received
0926 13:45:48.455100 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:48.455105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:48.455298 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:48.455784 (+ 486us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:48.455924 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.456777 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.458042 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.459042 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.459131 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:48.459265 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:48.459409 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 13:45:48.459491 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":130,"threads_started":1}
I20260926 13:45:48.459625 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:48.454297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:41831 (local address 127.0.0.1:51878)
0926 13:45:48.454560 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:48.454601 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:48.454706 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:48.455472 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:48.455478 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:48.455501 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:48.455611 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:48.455616 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.456884 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.456887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:48.457918 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:48.457927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.459144 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.459148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:48.459247 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:48.459282 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 13:45:48.459374 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 13:45:48.459484 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":1,"thread_start_us":61,"threads_started":1}
I20260926 13:45:48.460258 23702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:48.454908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:38093 (local address 127.0.0.1:33844)
0926 13:45:48.455398 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:48.455410 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:48.455481 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:48.456208 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:48.456211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:48.456227 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:48.456324 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:48.456331 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.457866 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.457869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:48.458890 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:48.458899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.459945 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.459948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:48.460040 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:48.460062 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 13:45:48.460126 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 13:45:48.460171 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":3,"thread_start_us":75,"threads_started":1}
I20260926 13:45:48.460426 23704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:48.455296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33844 (local address 127.12.170.65:38093)
0926 13:45:48.455835 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:48.455845 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:48.455863 (+ 18us) server_negotiation.cc:407] Connection header received
0926 13:45:48.455921 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:48.455925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:48.456076 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:48.456491 (+ 415us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:48.456643 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.457764 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.458995 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.459824 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.459922 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:48.460060 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:48.460152 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 13:45:48.460236 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":103,"threads_started":1}
I20260926 13:45:48.462553 23622 tablet_service.cc:1467] Processing CreateTablet for tablet e1cb2d6f90da485585cf1959a5d845bd (DEFAULT_TABLE table=test-table [id=9e6b34da1efd495e8a8ba4e66aaa7e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:48.463522 23622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1cb2d6f90da485585cf1959a5d845bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:48.466177 23701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:48.454597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:40117 (local address 127.0.0.1:49068)
0926 13:45:48.454902 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:48.454913 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:48.454984 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:48.456162 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:48.456165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:48.456183 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:48.456277 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:48.456282 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.457725 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.457728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:48.458925 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:48.458936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.465791 (+ 6855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.465794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:48.465886 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:48.465907 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 13:45:48.465978 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 13:45:48.466041 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":19,"thread_start_us":82,"threads_started":1}
I20260926 13:45:48.466226 23703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:48.454904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49068 (local address 127.12.170.66:40117)
0926 13:45:48.455752 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:48.455762 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:48.455784 (+ 22us) server_negotiation.cc:407] Connection header received
0926 13:45:48.455852 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:48.455858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:48.456032 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:48.456173 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:48.456413 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.457578 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.459024 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.465687 (+ 6663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.465779 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:48.465904 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:48.466003 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 13:45:48.466088 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":123,"threads_started":1}
I20260926 13:45:48.467512 23354 tablet_service.cc:1467] Processing CreateTablet for tablet e1cb2d6f90da485585cf1959a5d845bd (DEFAULT_TABLE table=test-table [id=9e6b34da1efd495e8a8ba4e66aaa7e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:48.468511 23354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1cb2d6f90da485585cf1959a5d845bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:48.469997 23705 tablet_bootstrap.cc:492] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0: Bootstrap starting.
I20260926 13:45:48.471796 23705 tablet_bootstrap.cc:654] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:48.473373 23705 log.cc:824] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:48.474781 23707 tablet_bootstrap.cc:492] T e1cb2d6f90da485585cf1959a5d845bd P f258fdc494224a638e10faf28d7014db: Bootstrap starting.
I20260926 13:45:48.474437 23477 tablet_service.cc:1467] Processing CreateTablet for tablet e1cb2d6f90da485585cf1959a5d845bd (DEFAULT_TABLE table=test-table [id=9e6b34da1efd495e8a8ba4e66aaa7e69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:45:48.475046 23705 tablet_bootstrap.cc:492] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0: No bootstrap required, opened a new log
I20260926 13:45:48.475225 23705 ts_tablet_manager.cc:1397] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260926 13:45:48.475392 23477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1cb2d6f90da485585cf1959a5d845bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:45:48.477146 23707 tablet_bootstrap.cc:654] T e1cb2d6f90da485585cf1959a5d845bd P f258fdc494224a638e10faf28d7014db: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:48.478341 23707 log.cc:824] T e1cb2d6f90da485585cf1959a5d845bd P f258fdc494224a638e10faf28d7014db: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:48.478883 23705 raft_consensus.cc:348] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd08b9c17a354084abff77df7048c0e0" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 41831 } } peers { permanent_uuid: "f258fdc494224a638e10faf28d7014db" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 38093 } } peers { permanent_uuid: "ed0bb488dbde4044858fb7ed2c381ed8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40117 } }
I20260926 13:45:48.479099 23705 raft_consensus.cc:374] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:48.479151 23705 raft_consensus.cc:729] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd08b9c17a354084abff77df7048c0e0, State: Initialized, Role: FOLLOWER
I20260926 13:45:48.479568 23705 consensus_queue.cc:260] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd08b9c17a354084abff77df7048c0e0" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 41831 } } peers { permanent_uuid: "f258fdc494224a638e10faf28d7014db" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 38093 } } peers { permanent_uuid: "ed0bb488dbde4044858fb7ed2c381ed8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40117 } }
I20260926 13:45:48.480190 23688 heartbeater.cc:503] Master 127.12.170.126:44561 was elected leader, sending a full tablet report...
I20260926 13:45:48.480229 23707 tablet_bootstrap.cc:492] T e1cb2d6f90da485585cf1959a5d845bd P f258fdc494224a638e10faf28d7014db: No bootstrap required, opened a new log
I20260926 13:45:48.480389 23707 ts_tablet_manager.cc:1397] T e1cb2d6f90da485585cf1959a5d845bd P f258fdc494224a638e10faf28d7014db: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260926 13:45:48.480522 23705 ts_tablet_manager.cc:1428] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 13:45:48.481801 23710 tablet_bootstrap.cc:492] T e1cb2d6f90da485585cf1959a5d845bd P ed0bb488dbde4044858fb7ed2c381ed8: Bootstrap starting.
I20260926 13:45:48.483749 23707 raft_consensus.cc:348] T e1cb2d6f90da485585cf1959a5d845bd P f258fdc494224a638e10faf28d7014db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd08b9c17a354084abff77df7048c0e0" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 41831 } } peers { permanent_uuid: "f258fdc494224a638e10faf28d7014db" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 38093 } } peers { permanent_uuid: "ed0bb488dbde4044858fb7ed2c381ed8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40117 } }
I20260926 13:45:48.483976 23707 raft_consensus.cc:374] T e1cb2d6f90da485585cf1959a5d845bd P f258fdc494224a638e10faf28d7014db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:48.484032 23707 raft_consensus.cc:729] T e1cb2d6f90da485585cf1959a5d845bd P f258fdc494224a638e10faf28d7014db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f258fdc494224a638e10faf28d7014db, State: Initialized, Role: FOLLOWER
I20260926 13:45:48.484151 23710 tablet_bootstrap.cc:654] T e1cb2d6f90da485585cf1959a5d845bd P ed0bb488dbde4044858fb7ed2c381ed8: Neither blocks nor log segments found. Creating new log.
I20260926 13:45:48.484797 23707 consensus_queue.cc:260] T e1cb2d6f90da485585cf1959a5d845bd P f258fdc494224a638e10faf28d7014db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd08b9c17a354084abff77df7048c0e0" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 41831 } } peers { permanent_uuid: "f258fdc494224a638e10faf28d7014db" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 38093 } } peers { permanent_uuid: "ed0bb488dbde4044858fb7ed2c381ed8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40117 } }
I20260926 13:45:48.484874 23710 log.cc:824] T e1cb2d6f90da485585cf1959a5d845bd P ed0bb488dbde4044858fb7ed2c381ed8: Log is configured to *not* fsync() on all Append() calls
I20260926 13:45:48.485351 23420 heartbeater.cc:503] Master 127.12.170.126:44561 was elected leader, sending a full tablet report...
I20260926 13:45:48.485601 23707 ts_tablet_manager.cc:1428] T e1cb2d6f90da485585cf1959a5d845bd P f258fdc494224a638e10faf28d7014db: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 13:45:48.486711 23710 tablet_bootstrap.cc:492] T e1cb2d6f90da485585cf1959a5d845bd P ed0bb488dbde4044858fb7ed2c381ed8: No bootstrap required, opened a new log
I20260926 13:45:48.486876 23710 ts_tablet_manager.cc:1397] T e1cb2d6f90da485585cf1959a5d845bd P ed0bb488dbde4044858fb7ed2c381ed8: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260926 13:45:48.489534 23710 raft_consensus.cc:348] T e1cb2d6f90da485585cf1959a5d845bd P ed0bb488dbde4044858fb7ed2c381ed8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd08b9c17a354084abff77df7048c0e0" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 41831 } } peers { permanent_uuid: "f258fdc494224a638e10faf28d7014db" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 38093 } } peers { permanent_uuid: "ed0bb488dbde4044858fb7ed2c381ed8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40117 } }
I20260926 13:45:48.489708 23710 raft_consensus.cc:374] T e1cb2d6f90da485585cf1959a5d845bd P ed0bb488dbde4044858fb7ed2c381ed8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:45:48.489758 23710 raft_consensus.cc:729] T e1cb2d6f90da485585cf1959a5d845bd P ed0bb488dbde4044858fb7ed2c381ed8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed0bb488dbde4044858fb7ed2c381ed8, State: Initialized, Role: FOLLOWER
I20260926 13:45:48.490103 23710 consensus_queue.cc:260] T e1cb2d6f90da485585cf1959a5d845bd P ed0bb488dbde4044858fb7ed2c381ed8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd08b9c17a354084abff77df7048c0e0" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 41831 } } peers { permanent_uuid: "f258fdc494224a638e10faf28d7014db" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 38093 } } peers { permanent_uuid: "ed0bb488dbde4044858fb7ed2c381ed8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40117 } }
I20260926 13:45:48.490633 23554 heartbeater.cc:503] Master 127.12.170.126:44561 was elected leader, sending a full tablet report...
I20260926 13:45:48.490869 23710 ts_tablet_manager.cc:1428] T e1cb2d6f90da485585cf1959a5d845bd P ed0bb488dbde4044858fb7ed2c381ed8: Time spent starting tablet: real 0.004s user 0.003s sys 0.002s
W20260926 13:45:48.494921 23421 tablet.cc:2369] T e1cb2d6f90da485585cf1959a5d845bd P f258fdc494224a638e10faf28d7014db: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:45:48.662776 23689 tablet.cc:2369] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:45:48.702744 23555 tablet.cc:2369] T e1cb2d6f90da485585cf1959a5d845bd P ed0bb488dbde4044858fb7ed2c381ed8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:45:48.731228 23709 raft_consensus.cc:482] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:45:48.731420 23709 raft_consensus.cc:504] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd08b9c17a354084abff77df7048c0e0" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 41831 } } peers { permanent_uuid: "f258fdc494224a638e10faf28d7014db" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 38093 } } peers { permanent_uuid: "ed0bb488dbde4044858fb7ed2c381ed8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40117 } }
I20260926 13:45:48.732601 23709 leader_election.cc:288] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f258fdc494224a638e10faf28d7014db (127.12.170.65:38093), ed0bb488dbde4044858fb7ed2c381ed8 (127.12.170.66:40117)
I20260926 13:45:48.736830 23703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:48.732913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:41445 (local address 127.12.170.66:40117)
0926 13:45:48.733038 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:48.733043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:48.733058 (+ 15us) server_negotiation.cc:407] Connection header received
0926 13:45:48.733120 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:48.733122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:48.733171 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:48.733242 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:48.733559 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.734243 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.735748 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.736280 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.736398 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:48.736448 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:48.736650 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 13:45:48.736716 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 13:45:48.736837 23690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:48.732835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:40117 (local address 127.12.170.67:41445)
0926 13:45:48.732934 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:48.732949 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:48.733036 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:48.733301 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:48.733304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:48.733330 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:48.733449 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:48.733457 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.734349 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.734351 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:48.735581 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:48.735593 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.736405 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.736409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:48.736519 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:48.736547 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 13:45:48.736622 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 13:45:48.736699 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 13:45:48.737105 23704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:48.732913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:35307 (local address 127.12.170.65:38093)
0926 13:45:48.733021 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:48.733024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:48.733509 (+ 485us) server_negotiation.cc:407] Connection header received
0926 13:45:48.733610 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:48.733613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:48.733662 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:48.733737 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:45:48.734059 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.734621 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.735970 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.736390 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.736523 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:48.736577 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:48.736889 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 13:45:48.736975 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:45:48.737424 23714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:48.732835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:38093 (local address 127.12.170.67:35307)
0926 13:45:48.733412 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:48.733425 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:45:48.733530 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:45:48.733784 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:45:48.733786 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:45:48.733835 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:45:48.733947 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:48.733954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.734732 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.734735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:48.735854 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:45:48.735861 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.736637 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.736640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:45:48.736741 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:48.736765 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 13:45:48.737249 (+ 484us) client_negotiation.cc:240] Negotiation successful
0926 13:45:48.737321 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":262,"thread_start_us":103,"threads_started":1}
I20260926 13:45:48.738035 23374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1cb2d6f90da485585cf1959a5d845bd" candidate_uuid: "dd08b9c17a354084abff77df7048c0e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f258fdc494224a638e10faf28d7014db" is_pre_election: true
I20260926 13:45:48.738087 23508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1cb2d6f90da485585cf1959a5d845bd" candidate_uuid: "dd08b9c17a354084abff77df7048c0e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed0bb488dbde4044858fb7ed2c381ed8" is_pre_election: true
I20260926 13:45:48.738317 23374 raft_consensus.cc:2454] T e1cb2d6f90da485585cf1959a5d845bd P f258fdc494224a638e10faf28d7014db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd08b9c17a354084abff77df7048c0e0 in term 0.
I20260926 13:45:48.738339 23508 raft_consensus.cc:2454] T e1cb2d6f90da485585cf1959a5d845bd P ed0bb488dbde4044858fb7ed2c381ed8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd08b9c17a354084abff77df7048c0e0 in term 0.
I20260926 13:45:48.738758 23575 leader_election.cc:302] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd08b9c17a354084abff77df7048c0e0, f258fdc494224a638e10faf28d7014db; no voters:
I20260926 13:45:48.739009 23709 raft_consensus.cc:2781] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:45:48.739095 23709 raft_consensus.cc:482] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:45:48.739136 23709 raft_consensus.cc:3037] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:48.740347 23709 raft_consensus.cc:504] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd08b9c17a354084abff77df7048c0e0" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 41831 } } peers { permanent_uuid: "f258fdc494224a638e10faf28d7014db" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 38093 } } peers { permanent_uuid: "ed0bb488dbde4044858fb7ed2c381ed8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40117 } }
I20260926 13:45:48.740674 23709 leader_election.cc:288] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0 [CANDIDATE]: Term 1 election: Requested vote from peers f258fdc494224a638e10faf28d7014db (127.12.170.65:38093), ed0bb488dbde4044858fb7ed2c381ed8 (127.12.170.66:40117)
I20260926 13:45:48.741078 23508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1cb2d6f90da485585cf1959a5d845bd" candidate_uuid: "dd08b9c17a354084abff77df7048c0e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed0bb488dbde4044858fb7ed2c381ed8"
I20260926 13:45:48.741122 23374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1cb2d6f90da485585cf1959a5d845bd" candidate_uuid: "dd08b9c17a354084abff77df7048c0e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f258fdc494224a638e10faf28d7014db"
I20260926 13:45:48.741199 23508 raft_consensus.cc:3037] T e1cb2d6f90da485585cf1959a5d845bd P ed0bb488dbde4044858fb7ed2c381ed8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:48.741225 23374 raft_consensus.cc:3037] T e1cb2d6f90da485585cf1959a5d845bd P f258fdc494224a638e10faf28d7014db [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:45:48.742460 23374 raft_consensus.cc:2454] T e1cb2d6f90da485585cf1959a5d845bd P f258fdc494224a638e10faf28d7014db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd08b9c17a354084abff77df7048c0e0 in term 1.
I20260926 13:45:48.742460 23508 raft_consensus.cc:2454] T e1cb2d6f90da485585cf1959a5d845bd P ed0bb488dbde4044858fb7ed2c381ed8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd08b9c17a354084abff77df7048c0e0 in term 1.
I20260926 13:45:48.742777 23576 leader_election.cc:302] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd08b9c17a354084abff77df7048c0e0, ed0bb488dbde4044858fb7ed2c381ed8; no voters:
I20260926 13:45:48.742960 23709 raft_consensus.cc:2781] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:45:48.743152 23709 raft_consensus.cc:686] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0 [term 1 LEADER]: Becoming Leader. State: Replica: dd08b9c17a354084abff77df7048c0e0, State: Running, Role: LEADER
I20260926 13:45:48.743449 23709 consensus_queue.cc:237] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd08b9c17a354084abff77df7048c0e0" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 41831 } } peers { permanent_uuid: "f258fdc494224a638e10faf28d7014db" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 38093 } } peers { permanent_uuid: "ed0bb488dbde4044858fb7ed2c381ed8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40117 } }
I20260926 13:45:48.746706 23228 catalog_manager.cc:5495] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd08b9c17a354084abff77df7048c0e0 (127.12.170.67). New cstate: current_term: 1 leader_uuid: "dd08b9c17a354084abff77df7048c0e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd08b9c17a354084abff77df7048c0e0" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 41831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f258fdc494224a638e10faf28d7014db" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 38093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed0bb488dbde4044858fb7ed2c381ed8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40117 } health_report { overall_health: UNKNOWN } } }
I20260926 13:45:48.820935 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:48.817835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51894 (local address 127.12.170.67:41831)
0926 13:45:48.817954 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 13:45:48.817958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:45:48.817970 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:45:48.818052 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:45:48.818054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:45:48.818106 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:45:48.818193 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:45:48.818375 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.818867 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.819635 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:45:48.819887 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:45:48.819977 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:45:48.820190 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:45:48.820588 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:45:48.820681 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 13:45:48.820709 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 13:45:48.820758 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:45:48.826581 23374 raft_consensus.cc:1261] T e1cb2d6f90da485585cf1959a5d845bd P f258fdc494224a638e10faf28d7014db [term 1 FOLLOWER]: Refusing update from remote peer dd08b9c17a354084abff77df7048c0e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:48.826592 23508 raft_consensus.cc:1261] T e1cb2d6f90da485585cf1959a5d845bd P ed0bb488dbde4044858fb7ed2c381ed8 [term 1 FOLLOWER]: Refusing update from remote peer dd08b9c17a354084abff77df7048c0e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:45:48.827275 23715 consensus_queue.cc:1035] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed0bb488dbde4044858fb7ed2c381ed8" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:48.827446 23709 consensus_queue.cc:1035] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f258fdc494224a638e10faf28d7014db" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 38093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:45:48.834636 23719 mvcc.cc:205] Tried to move back new op lower bound from 7333602708786995200 to 7333602708455829504. Current Snapshot: MvccSnapshot[applied={T|T < 7333602708786995200}]
I20260926 13:45:48.836336 23720 mvcc.cc:205] Tried to move back new op lower bound from 7333602708786995200 to 7333602708455829504. Current Snapshot: MvccSnapshot[applied={T|T < 7333602708786995200}]
I20260926 13:45:48.842161 23229 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:45:48.842303 23229 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:45:48.844527 23229 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21090777a1c54e3190a322aadec69429: Sending DeleteTablet for 3 replicas of tablet e1cb2d6f90da485585cf1959a5d845bd
I20260926 13:45:48.845273 23477 tablet_service.cc:1514] Processing DeleteTablet for tablet e1cb2d6f90da485585cf1959a5d845bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:48 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:49068
I20260926 13:45:48.845273 23622 tablet_service.cc:1514] Processing DeleteTablet for tablet e1cb2d6f90da485585cf1959a5d845bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:48 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:51878
I20260926 13:45:48.845367 23354 tablet_service.cc:1514] Processing DeleteTablet for tablet e1cb2d6f90da485585cf1959a5d845bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:45:48 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:33844
I20260926 13:45:48.845852 23729 tablet_replica.cc:330] T e1cb2d6f90da485585cf1959a5d845bd P ed0bb488dbde4044858fb7ed2c381ed8: stopping tablet replica
I20260926 13:45:48.845916 23730 tablet_replica.cc:330] T e1cb2d6f90da485585cf1959a5d845bd P f258fdc494224a638e10faf28d7014db: stopping tablet replica
I20260926 13:45:48.846189 23730 raft_consensus.cc:2229] T e1cb2d6f90da485585cf1959a5d845bd P f258fdc494224a638e10faf28d7014db [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:45:48.846216 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 23291
I20260926 13:45:48.846454 23730 raft_consensus.cc:2258] T e1cb2d6f90da485585cf1959a5d845bd P f258fdc494224a638e10faf28d7014db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:48.847707 23729 raft_consensus.cc:2229] T e1cb2d6f90da485585cf1959a5d845bd P ed0bb488dbde4044858fb7ed2c381ed8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:45:48.848048 23729 raft_consensus.cc:2258] T e1cb2d6f90da485585cf1959a5d845bd P ed0bb488dbde4044858fb7ed2c381ed8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:45:48.849290 23729 ts_tablet_manager.cc:1905] T e1cb2d6f90da485585cf1959a5d845bd P ed0bb488dbde4044858fb7ed2c381ed8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:45:48.851459 23729 ts_tablet_manager.cc:1918] T e1cb2d6f90da485585cf1959a5d845bd P ed0bb488dbde4044858fb7ed2c381ed8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:45:48.851557 23729 log.cc:1196] T e1cb2d6f90da485585cf1959a5d845bd P ed0bb488dbde4044858fb7ed2c381ed8: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-1/wal/wals/e1cb2d6f90da485585cf1959a5d845bd
I20260926 13:45:48.851835 23728 tablet_replica.cc:330] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0: stopping tablet replica
I20260926 13:45:48.851888 23729 ts_tablet_manager.cc:1939] T e1cb2d6f90da485585cf1959a5d845bd P ed0bb488dbde4044858fb7ed2c381ed8: Deleting consensus metadata
I20260926 13:45:48.852206 23728 raft_consensus.cc:2229] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:45:48.852632 23728 raft_consensus.cc:2258] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 13:45:48.852867 23213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.170.65:38093 (error 108)
I20260926 13:45:48.853329 23214 catalog_manager.cc:4841] TS ed0bb488dbde4044858fb7ed2c381ed8 (127.12.170.66:40117): tablet e1cb2d6f90da485585cf1959a5d845bd (table test-table [id=9e6b34da1efd495e8a8ba4e66aaa7e69]) successfully deleted
I20260926 13:45:48.853660 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 23425
I20260926 13:45:48.855358 23701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:45:48.855083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:38093 (local address 127.0.0.1:33848)
0926 13:45:48.855224 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 13:45:48.855283 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.65:38093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 13:45:48.855372 23728 ts_tablet_manager.cc:1905] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 13:45:48.855497 23213 catalog_manager.cc:4584] TS f258fdc494224a638e10faf28d7014db (127.12.170.65:38093): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e1cb2d6f90da485585cf1959a5d845bd: Network error: Client connection negotiation failed: client connection to 127.12.170.65:38093: connect: Connection refused (error 111)
I20260926 13:45:48.858206 23728 ts_tablet_manager.cc:1918] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:45:48.858317 23728 log.cc:1196] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/e1cb2d6f90da485585cf1959a5d845bd
I20260926 13:45:48.858655 23728 ts_tablet_manager.cc:1939] T e1cb2d6f90da485585cf1959a5d845bd P dd08b9c17a354084abff77df7048c0e0: Deleting consensus metadata
I20260926 13:45:48.859695 23213 catalog_manager.cc:4841] TS dd08b9c17a354084abff77df7048c0e0 (127.12.170.67:41831): tablet e1cb2d6f90da485585cf1959a5d845bd (table test-table [id=9e6b34da1efd495e8a8ba4e66aaa7e69]) successfully deleted
I20260926 13:45:48.859923 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 23559
I20260926 13:45:48.864446 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 23197
2026-09-26T13:45:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2341 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:45:48 dist-test-slave-hc1k krb5kdc[23734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:45:48 dist-test-slave-hc1k krb5kdc[23734](info): set up 2 sockets
Sep 26 13:45:48 dist-test-slave-hc1k krb5kdc[23734](info): commencing operation
krb5kdc: starting...
W20260926 13:45:49.816309 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:45:49 dist-test-slave-hc1k krb5kdc[23734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:45:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:45:49Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.170.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.170.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.170.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/rangeradmin_127.12.170.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.170.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/rangeradmin_127.12.170.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.170.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.170.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.170.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/rangerlookup_127.12.170.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.170.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/rangerlookup_127.12.170.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.170.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.170.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.170.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/HTTP_127.12.170.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.170.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/HTTP_127.12.170.84@KRBTEST.COM.keytab.
I20260926 13:45:49.963869 12969 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task0bSs4M/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/postgres -l logfile start
2026-09-26 13:45:52.177 UTC [23764] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 13:45:52.177 UTC [23764] LOG: listening on IPv4 address "127.12.170.84", port 38397
2026-09-26 13:45:52.180 UTC [23764] LOG: listening on Unix socket "/tmp/.s.PGSQL.38397"
2026-09-26 13:45:52.200 UTC [23767] LOG: database system was shut down at 2026-09-26 13:45:50 UTC
2026-09-26 13:45:52.204 UTC [23764] LOG: database system is ready to accept connections
I20260926 13:45:53.080214 12969 mini_postgres.cc:96] Postgres bound to 38397
2026-09-26 13:45:53.085 UTC [23775] FATAL: database "slave" does not exist
127.12.170.84:38397 - accepting connections
I20260926 13:45:53.086385 12969 mini_ranger.cc:161] Starting Ranger...
I20260926 13:45:53.099824 12969 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 13:45:53.503271 12969 mini_ranger.cc:87] Created ranger Postgres database
I20260926 13:45:53.503360 12969 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ranger-admin
2026-09-26 13:45:53,740 [I] DB FLAVOR :POSTGRES
2026-09-26 13:45:53,741 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 13:45:53,741 [I] Checking connection..
2026-09-26 13:45:53,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:38397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 13:45:53,954 [I] Checking connection passed.
2026-09-26 13:45:53,954 [I] --------- Verifying version history table ---------
2026-09-26 13:45:53,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:38397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 13:45:54,179 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 13:45:54,180 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:38397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 13:45:54,410 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 13:45:54,410 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 13:45:54,410 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:38397/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 13:45:54.650 UTC [23839] WARNING: there is no transaction in progress
2026-09-26 13:45:54,656 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 13:45:54,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:38397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 13:45:54,898 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 13:45:54,898 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 13:45:54,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:38397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 13:45:55,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:38397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 13:45:55,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:38397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hc1k.c.gcp-upstream.internal','N') ;"
2026-09-26 13:45:55,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:38397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 13:45:55,829 [I] Table x_portal_user does not exist in database ranger
2026-09-26 13:45:55,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:38397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 13:45:56,062 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 13:45:56,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:38397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 13:45:56,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:38397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 13:45:56,542 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 13:45:56,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:38397/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 13:45:57.418 UTC [23947] WARNING: there is no transaction in progress
2026-09-26 13:45:57.432 UTC [23947] WARNING: there is no transaction in progress
2026-09-26 13:45:57.446 UTC [23947] WARNING: there is no transaction in progress
2026-09-26 13:45:57.460 UTC [23947] WARNING: there is no transaction in progress
2026-09-26 13:45:57.474 UTC [23947] WARNING: there is no transaction in progress
2026-09-26 13:45:57.489 UTC [23947] WARNING: there is no transaction in progress
2026-09-26 13:45:57.537 UTC [23947] WARNING: there is no transaction in progress
2026-09-26 13:45:57.543 UTC [23947] WARNING: there is no transaction in progress
2026-09-26 13:45:57.555 UTC [23947] WARNING: there is no transaction in progress
2026-09-26 13:45:57.564 UTC [23947] WARNING: there is no transaction in progress
2026-09-26 13:45:57.573 UTC [23947] WARNING: there is no transaction in progress
2026-09-26 13:45:57.585 UTC [23947] WARNING: there is no transaction in progress
2026-09-26 13:45:57.594 UTC [23947] WARNING: there is no transaction in progress
2026-09-26 13:45:57.600 UTC [23947] WARNING: there is no transaction in progress
2026-09-26 13:45:57.605 UTC [23947] WARNING: there is no transaction in progress
2026-09-26 13:45:57.611 UTC [23947] WARNING: there is no transaction in progress
2026-09-26 13:45:58,171 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 13:45:58,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:38397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 13:45:58,413 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 13:45:58,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:38397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 13:45:58,639 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 13:45:58,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:38397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 13:45:58,866 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 13:45:58,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:38397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 13:45:59,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:38397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 13:45:59,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:38397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hc1k.c.gcp-upstream.internal';"
2026-09-26 13:45:59,556 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 13:45:59,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:38397/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 13:45:59,783 [I] DB_PATCHES have already been applied
I20260926 13:45:59.788938 12969 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ranger-admin:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task0bSs4M/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 13:45:59.788995 12969 mini_ranger.cc:193] Using host: 127.12.170.84
I20260926 13:45:59.792074 12969 mini_ranger.cc:227] Ranger admin URL: http://127.12.170.84:33177
Sep 26, 2026 1:46:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:46:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:46:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 1:46:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 1:46:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 1:46:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 1:46:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 13:46:00.771148 12969 mini_ranger.cc:160] Time spent starting Ranger: real 7.685s user 0.000s sys 0.008s
Sep 26, 2026 1:46:18 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 1:46:18 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 1:46:18 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 1:46:18 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 1:46:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 1:46:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 1:46:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 1:46:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 1:46:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 1:46:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 1:46:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 1:46:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 1:46:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 1:46:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 1:46:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 1:46:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 1:46:18 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 1:46:19 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 13:46:20.091126 12969 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.170.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.170.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.170.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/rangerkms_127.12.170.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.170.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/rangerkms_127.12.170.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.170.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.170.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.170.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/HTTP_127.12.170.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.170.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/HTTP_127.12.170.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 13:46:20 dist-test-slave-hc1k krb5kdc[23734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430380, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 13:46:20.184795 12969 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 13:46:20.193809 12969 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 13:46:21.000265 12969 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 13:46:21.000342 12969 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ranger-kms
I20260926 13:46:21.000433 12969 mini_ranger_kms.cc:227] Using postgres at 127.12.170.84:38397
2026-09-26 13:46:21,164 [I] DB FLAVOR :POSTGRES
2026-09-26 13:46:21,164 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 13:46:21,164 [I] Checking connection
2026-09-26 13:46:21,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:38397/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 13:46:21,385 [I] connection success
2026-09-26 13:46:21,385 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 13:46:21,386 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 13:46:21,386 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:38397/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 13:46:21,614 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 13:46:21,614 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 13:46:21,614 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 13:46:21,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:38397/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 13:46:21,857 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 13:46:22.305476 12969 mini_ranger_kms.cc:310] Created kms service
I20260926 13:46:22.511282 12969 mini_ranger_kms.cc:326] Created kudu user
I20260926 13:46:22.555634 12969 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 13:46:22 dist-test-slave-hc1k krb5kdc[23734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430380, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.170.84@KRBTEST.COM
I20260926 13:46:22.926046 12969 mini_ranger_kms.cc:384] Added ranger policy
I20260926 13:46:22.926211 12969 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ranger-kms:/tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 13:46:22.926234 12969 mini_ranger_kms.cc:240] Using host: 127.12.170.84
I20260926 13:46:22.930362 12969 mini_ranger_kms.cc:276] Ranger KMS PID: 24127
I20260926 13:46:22.930495 12969 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.170.84:40819
Sep 26, 2026 1:46:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:46:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:46:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 1:46:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 1:46:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 1:46:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 13:46:25.019037 12969 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.834s user 0.010s sys 0.138s
I20260926 13:46:25.019170 12969 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 13:46:25.019284 12969 mini_ranger_kms.cc:401] 127.12.170.84:40819/kms/v1/keys
Sep 26 13:46:25 dist-test-slave-hc1k krb5kdc[23734](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790430385, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.170.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 13:46:25 dist-test-slave-hc1k krb5kdc[23734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430385, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.170.84@KRBTEST.COM for HTTP/127.12.170.84@KRBTEST.COM
Sep 26 13:46:25 dist-test-slave-hc1k krb5kdc[23734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430385, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.170.84@KRBTEST.COM for HTTP/127.12.170.84@KRBTEST.COM
Sep 26 13:46:27 dist-test-slave-hc1k krb5kdc[23734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:46:27.982008 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:45155
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:45155
--ranger_config_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.170.84:40819/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:46:28.089596 24178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:46:28.089851 24178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:46:28.089905 24178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:46:28.093819 24178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:46:28.093874 24178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:46:28.093900 24178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:46:28.093921 24178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:46:28.093946 24178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:46:28.099265 24178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.170.84:40819/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:45155
--ranger_config_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:45155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:46:28.100252 24178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:46:28.101178 24178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:46:28.110339 24183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:46:28.110450 24184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:46:28.110523 24178 server_base.cc:1034] running on GCE node
W20260926 13:46:28.110570 24186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:46:28.111183 24178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:46:28.112239 24178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:46:28.113399 24178 hybrid_clock.cc:648] HybridClock initialized: now 1790430388113376 us; error 39 us; skew 500 ppm
Sep 26 13:46:28 dist-test-slave-hc1k krb5kdc[23734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430388, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:46:28.119092 24178 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:46:28.120127 24178 webserver.cc:460] Webserver started at http://127.12.170.126:40413/ using document root <none> and password file <none>
I20260926 13:46:28.120743 24178 fs_manager.cc:359] Metadata directory not provided
I20260926 13:46:28.120812 24178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:46:28.121013 24178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:46:28 dist-test-slave-hc1k krb5kdc[23734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430388, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for HTTP/127.12.170.84@KRBTEST.COM
I20260926 13:46:28.166834 24178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "0a7fe52d5f1d4396808a1e012cd5a2b7"
format_stamp: "Formatted at 2026-09-26 13:46:28 on dist-test-slave-hc1k"
server_key: "680b0f32effae37b096ba555d9a608d6"
server_key_iv: "e2878568747bc9b95d70249bd726eb7a"
server_key_version: "kuduclusterkey@0"
I20260926 13:46:28.167395 24178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "0a7fe52d5f1d4396808a1e012cd5a2b7"
format_stamp: "Formatted at 2026-09-26 13:46:28 on dist-test-slave-hc1k"
server_key: "680b0f32effae37b096ba555d9a608d6"
server_key_iv: "e2878568747bc9b95d70249bd726eb7a"
server_key_version: "kuduclusterkey@0"
I20260926 13:46:28.171031 24178 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 13:46:28.187398 24193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:46:28.188723 24178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:46:28.188831 24178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "0a7fe52d5f1d4396808a1e012cd5a2b7"
format_stamp: "Formatted at 2026-09-26 13:46:28 on dist-test-slave-hc1k"
server_key: "680b0f32effae37b096ba555d9a608d6"
server_key_iv: "e2878568747bc9b95d70249bd726eb7a"
server_key_version: "kuduclusterkey@0"
I20260926 13:46:28.188957 24178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:46:28.222886 24178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:46:28.225361 24178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:46:28.225528 24178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:46:28.232403 24178 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:45155
I20260926 13:46:28.232419 24245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:45155 every 8 connection(s)
I20260926 13:46:28.233325 24178 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:46:28.235996 24246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:46:28.238816 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 24178
I20260926 13:46:28.238919 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:46:28.239140 12969 external_mini_cluster.cc:1428] Setting key 42212518c5d0c95123418f7ff38c22fc
I20260926 13:46:28.241736 24246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a7fe52d5f1d4396808a1e012cd5a2b7: Bootstrap starting.
I20260926 13:46:28.244328 24246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a7fe52d5f1d4396808a1e012cd5a2b7: Neither blocks nor log segments found. Creating new log.
I20260926 13:46:28.244985 24246 log.cc:824] T 00000000000000000000000000000000 P 0a7fe52d5f1d4396808a1e012cd5a2b7: Log is configured to *not* fsync() on all Append() calls
I20260926 13:46:28.247103 24246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a7fe52d5f1d4396808a1e012cd5a2b7: No bootstrap required, opened a new log
Sep 26 13:46:28 dist-test-slave-hc1k krb5kdc[23734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:46:28.250412 24246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a7fe52d5f1d4396808a1e012cd5a2b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7fe52d5f1d4396808a1e012cd5a2b7" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45155 } }
I20260926 13:46:28.250626 24246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a7fe52d5f1d4396808a1e012cd5a2b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:46:28.250686 24246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a7fe52d5f1d4396808a1e012cd5a2b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a7fe52d5f1d4396808a1e012cd5a2b7, State: Initialized, Role: FOLLOWER
I20260926 13:46:28.251098 24246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a7fe52d5f1d4396808a1e012cd5a2b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7fe52d5f1d4396808a1e012cd5a2b7" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45155 } }
I20260926 13:46:28.251222 24246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a7fe52d5f1d4396808a1e012cd5a2b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:46:28.251298 24246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a7fe52d5f1d4396808a1e012cd5a2b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:46:28.251389 24246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a7fe52d5f1d4396808a1e012cd5a2b7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:46:28.252144 24249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:46:28.240467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35932 (local address 127.12.170.126:45155)
0926 13:46:28.240904 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0926 13:46:28.240917 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 13:46:28.240951 (+ 34us) server_negotiation.cc:407] Connection header received
0926 13:46:28.241537 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:46:28.241553 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:46:28.241893 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:46:28.242226 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:46:28.242968 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:46:28.243972 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:46:28.244858 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:46:28.245355 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:46:28.245505 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:46:28.248073 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:46:28.248078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:46:28.248090 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:46:28.248115 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:46:28.249550 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:46:28.249952 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:46:28.249956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:46:28.249960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:46:28.250056 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:46:28.250312 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:46:28.250315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:46:28.250317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:46:28.250692 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:46:28.250879 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 13:46:28.251146 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 13:46:28.251427 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":147,"threads_started":1}
I20260926 13:46:28.252231 24246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a7fe52d5f1d4396808a1e012cd5a2b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7fe52d5f1d4396808a1e012cd5a2b7" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45155 } }
I20260926 13:46:28.252548 24246 leader_election.cc:302] T 00000000000000000000000000000000 P 0a7fe52d5f1d4396808a1e012cd5a2b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a7fe52d5f1d4396808a1e012cd5a2b7; no voters:
I20260926 13:46:28.252820 24246 leader_election.cc:288] T 00000000000000000000000000000000 P 0a7fe52d5f1d4396808a1e012cd5a2b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:46:28.252894 24251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a7fe52d5f1d4396808a1e012cd5a2b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:46:28.253085 24251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a7fe52d5f1d4396808a1e012cd5a2b7 [term 1 LEADER]: Becoming Leader. State: Replica: 0a7fe52d5f1d4396808a1e012cd5a2b7, State: Running, Role: LEADER
I20260926 13:46:28.253436 24251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a7fe52d5f1d4396808a1e012cd5a2b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7fe52d5f1d4396808a1e012cd5a2b7" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45155 } }
I20260926 13:46:28.254053 24246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a7fe52d5f1d4396808a1e012cd5a2b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:46:28.254566 24252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a7fe52d5f1d4396808a1e012cd5a2b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a7fe52d5f1d4396808a1e012cd5a2b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7fe52d5f1d4396808a1e012cd5a2b7" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45155 } } }
I20260926 13:46:28.254594 24253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a7fe52d5f1d4396808a1e012cd5a2b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a7fe52d5f1d4396808a1e012cd5a2b7. Latest consensus state: current_term: 1 leader_uuid: "0a7fe52d5f1d4396808a1e012cd5a2b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a7fe52d5f1d4396808a1e012cd5a2b7" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45155 } } }
I20260926 13:46:28.254678 24252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a7fe52d5f1d4396808a1e012cd5a2b7 [sys.catalog]: This master's current role is: LEADER
I20260926 13:46:28.254683 24253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a7fe52d5f1d4396808a1e012cd5a2b7 [sys.catalog]: This master's current role is: LEADER
I20260926 13:46:28.255981 24256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:46:28.256243 24246 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 13:46:28.261428 24256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:46:28.267165 24256 catalog_manager.cc:1347] Generated new cluster ID: 3c46b1502a4449a6bc43dffadda9d78f
I20260926 13:46:28.267232 24256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:46:28.282423 24256 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:46:28.283339 24256 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:46:28.292133 24256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a7fe52d5f1d4396808a1e012cd5a2b7: Generated new TSK 0
I20260926 13:46:28.292877 24256 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.170.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
I20260926 13:46:28.960703 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.170.84:40819/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:0
--local_ip_for_outbound_sockets=127.12.170.65
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:45155
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:46:29.101342 24291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:46:29.101706 24291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:46:29.101773 24291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:46:29.108098 24291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:46:29.108222 24291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:46:29.108331 24291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:46:29.115994 24291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.170.84:40819/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:45155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:46:29.117254 24291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:46:29.118590 24291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:46:29.130400 24301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:46:29.130470 24298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:46:29.130384 24299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:46:29.130631 24291 server_base.cc:1034] running on GCE node
I20260926 13:46:29.131233 24291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:46:29.131963 24291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:46:29.133153 24291 hybrid_clock.cc:648] HybridClock initialized: now 1790430389133132 us; error 43 us; skew 500 ppm
Sep 26 13:46:29 dist-test-slave-hc1k krb5kdc[23734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430389, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:46:29.137356 24291 init.cc:377] Logged in from keytab as kudu/127.12.170.65@KRBTEST.COM (short username kudu)
I20260926 13:46:29.138492 24291 webserver.cc:460] Webserver started at http://127.12.170.65:36565/ using document root <none> and password file <none>
I20260926 13:46:29.139176 24291 fs_manager.cc:359] Metadata directory not provided
I20260926 13:46:29.139262 24291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:46:29.139494 24291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:46:29 dist-test-slave-hc1k krb5kdc[23734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430389, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for HTTP/127.12.170.84@KRBTEST.COM
I20260926 13:46:29.167897 24291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-0/data/instance:
uuid: "afcd7a87b05d474e9520461b091cb73c"
format_stamp: "Formatted at 2026-09-26 13:46:29 on dist-test-slave-hc1k"
server_key: "d4d4771f31ba3a14830028849e13ccf7"
server_key_iv: "e60472158b5b2481db2b988fd5804127"
server_key_version: "kuduclusterkey@0"
I20260926 13:46:29.168731 24291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance:
uuid: "afcd7a87b05d474e9520461b091cb73c"
format_stamp: "Formatted at 2026-09-26 13:46:29 on dist-test-slave-hc1k"
server_key: "d4d4771f31ba3a14830028849e13ccf7"
server_key_iv: "e60472158b5b2481db2b988fd5804127"
server_key_version: "kuduclusterkey@0"
I20260926 13:46:29.183705 24291 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.004s sys 0.000s
I20260926 13:46:29.204698 24310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:46:29.206406 24291 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260926 13:46:29.206532 24291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "afcd7a87b05d474e9520461b091cb73c"
format_stamp: "Formatted at 2026-09-26 13:46:29 on dist-test-slave-hc1k"
server_key: "d4d4771f31ba3a14830028849e13ccf7"
server_key_iv: "e60472158b5b2481db2b988fd5804127"
server_key_version: "kuduclusterkey@0"
I20260926 13:46:29.206662 24291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:46:29.239683 24291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:46:29.243479 24291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:46:29.243736 24291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:46:29.244462 24291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:46:29.245502 24291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:46:29.245599 24291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:46:29.245685 24291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:46:29.245741 24291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 13:46:29 dist-test-slave-hc1k krb5kdc[23734](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790430389, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:46:29.256969 24291 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:39825
I20260926 13:46:29.258137 24291 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:46:29.259136 24423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:39825 every 8 connection(s)
I20260926 13:46:29.268175 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 24291
I20260926 13:46:29.268298 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance
I20260926 13:46:29.268597 12969 external_mini_cluster.cc:1428] Setting key fefe5d351b90103ea92a02aeb439e6dd
Sep 26 13:46:29 dist-test-slave-hc1k krb5kdc[23734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430389, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:46:29.277966 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:46:29.261719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:37655 (local address 127.12.170.126:45155)
0926 13:46:29.261997 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0926 13:46:29.262004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:46:29.265349 (+ 3345us) server_negotiation.cc:407] Connection header received
0926 13:46:29.265403 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:46:29.265406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:46:29.265462 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:46:29.265548 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:46:29.267006 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:46:29.267884 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:46:29.269564 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:46:29.269923 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:46:29.270034 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:46:29.273713 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:46:29.273717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:46:29.273720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:46:29.273747 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:46:29.275555 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:46:29.276199 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:46:29.276203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:46:29.276205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:46:29.276293 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:46:29.276680 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:46:29.276683 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:46:29.276685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:46:29.276902 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:46:29.276985 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 13:46:29.277658 (+ 673us) server_negotiation.cc:299] Negotiation successful
0926 13:46:29.277789 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":107,"threads_started":1}
I20260926 13:46:29.279887 24427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:46:29.262276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:45155 (local address 127.12.170.65:37655)
0926 13:46:29.263975 (+ 1699us) negotiation.cc:107] Waiting for socket to connect
0926 13:46:29.264060 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0926 13:46:29.265055 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:46:29.265852 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:46:29.265861 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:46:29.266207 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:46:29.266841 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:46:29.266858 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:46:29.268042 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:46:29.268046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:46:29.269422 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:46:29.269433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:46:29.270202 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:46:29.270206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:46:29.270396 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:46:29.271760 (+ 1364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:46:29.271781 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:46:29.273495 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:46:29.275731 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:46:29.275737 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:46:29.275754 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:46:29.276087 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:46:29.276387 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:46:29.276390 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:46:29.276392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:46:29.276561 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:46:29.277038 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:46:29.277045 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:46:29.277358 (+ 313us) client_negotiation.cc:769] Sending connection context
0926 13:46:29.278443 (+ 1085us) client_negotiation.cc:240] Negotiation successful
0926 13:46:29.278793 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1474,"thread_start_us":155,"threads_started":1}
I20260926 13:46:29.281522 24424 heartbeater.cc:347] Connected to a master server at 127.12.170.126:45155
I20260926 13:46:29.281775 24424 heartbeater.cc:464] Registering TS with master...
I20260926 13:46:29.282331 24424 heartbeater.cc:511] Master 127.12.170.126:45155 requested a full tablet report, sending...
I20260926 13:46:29.284200 24210 ts_manager.cc:194] Registered new tserver with Master: afcd7a87b05d474e9520461b091cb73c (127.12.170.65:39825)
I20260926 13:46:29.285771 24210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.65@KRBTEST.COM'} at 127.12.170.65:37655
WARNING: no policy specified for kudu/127.12.170.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
I20260926 13:46:29.309092 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.170.84:40819/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:0
--local_ip_for_outbound_sockets=127.12.170.66
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:45155
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:46:29.469281 24430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:46:29.469637 24430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:46:29.469705 24430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:46:29.475456 24430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:46:29.475535 24430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:46:29.475625 24430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:46:29.483302 24430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.170.84:40819/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:45155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:46:29.484706 24430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:46:29.486111 24430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:46:29.499063 24437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:46:29.499501 24436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:46:29.499842 24439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:46:29.500939 24430 server_base.cc:1034] running on GCE node
I20260926 13:46:29.501420 24430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:46:29.502170 24430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:46:29.503351 24430 hybrid_clock.cc:648] HybridClock initialized: now 1790430389503333 us; error 35 us; skew 500 ppm
Sep 26 13:46:29 dist-test-slave-hc1k krb5kdc[23734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430389, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:46:29.506778 24430 init.cc:377] Logged in from keytab as kudu/127.12.170.66@KRBTEST.COM (short username kudu)
I20260926 13:46:29.507747 24430 webserver.cc:460] Webserver started at http://127.12.170.66:36999/ using document root <none> and password file <none>
I20260926 13:46:29.508273 24430 fs_manager.cc:359] Metadata directory not provided
I20260926 13:46:29.508337 24430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:46:29.508538 24430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:46:29 dist-test-slave-hc1k krb5kdc[23734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430389, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for HTTP/127.12.170.84@KRBTEST.COM
I20260926 13:46:29.522927 24430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-1/data/instance:
uuid: "f4cb1eac10c14954b295429379dfa711"
format_stamp: "Formatted at 2026-09-26 13:46:29 on dist-test-slave-hc1k"
server_key: "922532322d46219563f124c685bd1315"
server_key_iv: "496e252892987638f51c5ef83958f297"
server_key_version: "kuduclusterkey@0"
I20260926 13:46:29.523484 24430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance:
uuid: "f4cb1eac10c14954b295429379dfa711"
format_stamp: "Formatted at 2026-09-26 13:46:29 on dist-test-slave-hc1k"
server_key: "922532322d46219563f124c685bd1315"
server_key_iv: "496e252892987638f51c5ef83958f297"
server_key_version: "kuduclusterkey@0"
I20260926 13:46:29.527336 24430 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 13:46:29.542003 24446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:46:29.543331 24430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:46:29.543444 24430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "f4cb1eac10c14954b295429379dfa711"
format_stamp: "Formatted at 2026-09-26 13:46:29 on dist-test-slave-hc1k"
server_key: "922532322d46219563f124c685bd1315"
server_key_iv: "496e252892987638f51c5ef83958f297"
server_key_version: "kuduclusterkey@0"
I20260926 13:46:29.543558 24430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:46:29.556228 24430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:46:29.559083 24430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:46:29.559240 24430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:46:29.559813 24430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:46:29.560642 24430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:46:29.560688 24430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:46:29.560724 24430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:46:29.560750 24430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:46:29.570817 24430 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:43457
I20260926 13:46:29.571771 24559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:43457 every 8 connection(s)
I20260926 13:46:29.571899 24430 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:46:29.579559 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 24430
I20260926 13:46:29.579691 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance
I20260926 13:46:29.579974 12969 external_mini_cluster.cc:1428] Setting key b80f1818076c0bbf49db0eecaf97393f
Sep 26 13:46:29 dist-test-slave-hc1k krb5kdc[23734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430389, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:46:29.588649 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:46:29.575833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:38861 (local address 127.12.170.126:45155)
0926 13:46:29.575944 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 13:46:29.575948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:46:29.575963 (+ 15us) server_negotiation.cc:407] Connection header received
0926 13:46:29.576967 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:46:29.576972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:46:29.577030 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:46:29.577077 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:46:29.578232 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:46:29.578742 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:46:29.580099 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:46:29.580401 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:46:29.580473 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:46:29.584666 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:46:29.584670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:46:29.584672 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:46:29.584697 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:46:29.586328 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:46:29.586911 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:46:29.586914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:46:29.586916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:46:29.587002 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:46:29.587395 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:46:29.587398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:46:29.587400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:46:29.587610 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:46:29.587724 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 13:46:29.588369 (+ 645us) server_negotiation.cc:299] Negotiation successful
0926 13:46:29.588489 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
WARNING: no policy specified for kudu/127.12.170.67@KRBTEST.COM; defaulting to no policy
I20260926 13:46:29.590317 24562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:46:29.574935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:45155 (local address 127.12.170.66:38861)
0926 13:46:29.575546 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0926 13:46:29.575608 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 13:46:29.576689 (+ 1081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:46:29.577394 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:46:29.577400 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:46:29.577653 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:46:29.578096 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:46:29.578109 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:46:29.578854 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:46:29.578858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:46:29.579963 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:46:29.579973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:46:29.580656 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:46:29.580660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:46:29.580834 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:46:29.582847 (+ 2013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:46:29.582867 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:46:29.584494 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:46:29.586457 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:46:29.586463 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:46:29.586479 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:46:29.586793 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:46:29.587121 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:46:29.587123 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:46:29.587125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:46:29.587293 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:46:29.587786 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:46:29.587792 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:46:29.588095 (+ 303us) client_negotiation.cc:769] Sending connection context
0926 13:46:29.589064 (+ 969us) client_negotiation.cc:240] Negotiation successful
0926 13:46:29.589389 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":127,"threads_started":1}
I20260926 13:46:29.591482 24560 heartbeater.cc:347] Connected to a master server at 127.12.170.126:45155
I20260926 13:46:29.591761 24560 heartbeater.cc:464] Registering TS with master...
I20260926 13:46:29.592355 24560 heartbeater.cc:511] Master 127.12.170.126:45155 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.67@KRBTEST.COM" created.
I20260926 13:46:29.593554 24210 ts_manager.cc:194] Registered new tserver with Master: f4cb1eac10c14954b295429379dfa711 (127.12.170.66:43457)
I20260926 13:46:29.594225 24210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.66@KRBTEST.COM'} at 127.12.170.66:38861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
I20260926 13:46:29.614758 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.170.84:40819/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:0
--local_ip_for_outbound_sockets=127.12.170.67
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:45155
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:46:29 dist-test-slave-hc1k krb5kdc[23734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430389, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for HTTP/127.12.170.84@KRBTEST.COM
W20260926 13:46:29.751559 24565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:46:29.751946 24565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:46:29.752024 24565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:46:29.757464 24565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:46:29.757530 24565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:46:29.757624 24565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:46:29.764925 24565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.170.84:40819/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:45155
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:46:29.766296 24565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:46:29.767637 24565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:46:29.781028 24573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:46:29.781123 24570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:46:29.781023 24571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:46:29.782294 24565 server_base.cc:1034] running on GCE node
I20260926 13:46:29.782814 24565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:46:29.783623 24565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:46:29.784806 24565 hybrid_clock.cc:648] HybridClock initialized: now 1790430389784739 us; error 82 us; skew 500 ppm
Sep 26 13:46:29 dist-test-slave-hc1k krb5kdc[23734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430389, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:46:29.788331 24565 init.cc:377] Logged in from keytab as kudu/127.12.170.67@KRBTEST.COM (short username kudu)
I20260926 13:46:29.789390 24565 webserver.cc:460] Webserver started at http://127.12.170.67:41971/ using document root <none> and password file <none>
I20260926 13:46:29.790087 24565 fs_manager.cc:359] Metadata directory not provided
I20260926 13:46:29.790164 24565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:46:29.790388 24565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:46:29 dist-test-slave-hc1k krb5kdc[23734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430389, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for HTTP/127.12.170.84@KRBTEST.COM
I20260926 13:46:29.811239 24565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-2/data/instance:
uuid: "71d5e7d42f1a485d895d865c04bf7263"
format_stamp: "Formatted at 2026-09-26 13:46:29 on dist-test-slave-hc1k"
server_key: "93725042f299b12cadae3354fc5bbfd7"
server_key_iv: "72d3b8999e3278dad29b7d0f579f2549"
server_key_version: "kuduclusterkey@0"
I20260926 13:46:29.812052 24565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance:
uuid: "71d5e7d42f1a485d895d865c04bf7263"
format_stamp: "Formatted at 2026-09-26 13:46:29 on dist-test-slave-hc1k"
server_key: "93725042f299b12cadae3354fc5bbfd7"
server_key_iv: "72d3b8999e3278dad29b7d0f579f2549"
server_key_version: "kuduclusterkey@0"
I20260926 13:46:29.816768 24565 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 13:46:29.840142 24580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:46:29.841908 24565 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260926 13:46:29.842033 24565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "71d5e7d42f1a485d895d865c04bf7263"
format_stamp: "Formatted at 2026-09-26 13:46:29 on dist-test-slave-hc1k"
server_key: "93725042f299b12cadae3354fc5bbfd7"
server_key_iv: "72d3b8999e3278dad29b7d0f579f2549"
server_key_version: "kuduclusterkey@0"
I20260926 13:46:29.842151 24565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:46:29.869758 24565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:46:29.873353 24565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:46:29.873607 24565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:46:29.874354 24565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:46:29.875403 24565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:46:29.875505 24565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:46:29.875609 24565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:46:29.875694 24565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:46:29.885228 24565 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:33533
I20260926 13:46:29.886090 24693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:33533 every 8 connection(s)
I20260926 13:46:29.886587 24565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:46:29.892906 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 24565
I20260926 13:46:29.893016 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance
I20260926 13:46:29.893311 12969 external_mini_cluster.cc:1428] Setting key b9587a68d8b39b068784197ed67195fd
Sep 26 13:46:29 dist-test-slave-hc1k krb5kdc[23734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430389, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:46:29.902096 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:46:29.888579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:50441 (local address 127.12.170.126:45155)
0926 13:46:29.888702 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 13:46:29.888707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:46:29.889753 (+ 1046us) server_negotiation.cc:407] Connection header received
0926 13:46:29.890974 (+ 1221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:46:29.890979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:46:29.891032 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:46:29.891118 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:46:29.892493 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:46:29.893286 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:46:29.894981 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:46:29.895328 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:46:29.895408 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:46:29.898255 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:46:29.898260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:46:29.898263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:46:29.898288 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:46:29.899949 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:46:29.900525 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:46:29.900529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:46:29.900531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:46:29.900615 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:46:29.900970 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:46:29.900973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:46:29.900975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:46:29.901178 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:46:29.901274 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 13:46:29.901828 (+ 554us) server_negotiation.cc:299] Negotiation successful
0926 13:46:29.901942 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 13:46:29.902988 24696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:46:29.889057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:45155 (local address 127.12.170.67:50441)
0926 13:46:29.889577 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 13:46:29.889645 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 13:46:29.890733 (+ 1088us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:46:29.891317 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:46:29.891326 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:46:29.891699 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:46:29.892337 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:46:29.892359 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:46:29.893450 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:46:29.893453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:46:29.894840 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:46:29.894851 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:46:29.895610 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:46:29.895614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:46:29.895849 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:46:29.896467 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:46:29.896487 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:46:29.898091 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:46:29.900092 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:46:29.900097 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:46:29.900111 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:46:29.900431 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:46:29.900705 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:46:29.900708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:46:29.900710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:46:29.900879 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:46:29.901283 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:46:29.901290 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:46:29.901562 (+ 272us) client_negotiation.cc:769] Sending connection context
0926 13:46:29.901773 (+ 211us) client_negotiation.cc:240] Negotiation successful
0926 13:46:29.902061 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":126,"threads_started":1}
I20260926 13:46:29.904181 24694 heartbeater.cc:347] Connected to a master server at 127.12.170.126:45155
I20260926 13:46:29.904412 24694 heartbeater.cc:464] Registering TS with master...
I20260926 13:46:29.904943 24694 heartbeater.cc:511] Master 127.12.170.126:45155 requested a full tablet report, sending...
I20260926 13:46:29.906311 24209 ts_manager.cc:194] Registered new tserver with Master: 71d5e7d42f1a485d895d865c04bf7263 (127.12.170.67:33533)
I20260926 13:46:29.906977 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:46:29.907138 24209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.67@KRBTEST.COM'} at 127.12.170.67:50441
I20260926 13:46:29.920717 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:46:29.912059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35938 (local address 127.12.170.126:45155)
0926 13:46:29.912192 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 13:46:29.912196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:46:29.912472 (+ 276us) server_negotiation.cc:407] Connection header received
0926 13:46:29.912575 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:46:29.912578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:46:29.912631 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:46:29.912717 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:46:29.913432 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:46:29.914212 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:46:29.915311 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:46:29.915678 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:46:29.915784 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:46:29.917355 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:46:29.917359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:46:29.917362 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:46:29.917384 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:46:29.918883 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:46:29.919328 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:46:29.919331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:46:29.919333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:46:29.919405 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:46:29.919701 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:46:29.919704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:46:29.919706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:46:29.919898 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:46:29.920022 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 13:46:29.920275 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 13:46:29.920386 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 13:46:29.924121 24209 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:46:29.928531 24209 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:46:29.947808 24708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:46:29.937786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:43457 (local address 127.0.0.1:60184)
0926 13:46:29.940409 (+ 2623us) negotiation.cc:107] Waiting for socket to connect
0926 13:46:29.940459 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 13:46:29.940583 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:46:29.942254 (+ 1671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:46:29.942260 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:46:29.942282 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:46:29.942429 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:46:29.942436 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:46:29.943899 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:46:29.943903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:46:29.945294 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:46:29.945305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:46:29.947191 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:46:29.947195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:46:29.947322 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:46:29.947373 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 13:46:29.947463 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 13:46:29.947615 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2450,"thread_start_us":89,"threads_started":1}
I20260926 13:46:29.951239 24710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:46:29.940204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:39825 (local address 127.0.0.1:56338)
0926 13:46:29.945510 (+ 5306us) negotiation.cc:107] Waiting for socket to connect
0926 13:46:29.945523 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:46:29.945603 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:46:29.946428 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:46:29.946431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:46:29.946447 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:46:29.946535 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:46:29.946540 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:46:29.948108 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:46:29.948111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:46:29.949530 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:46:29.949540 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:46:29.950850 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:46:29.950854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:46:29.950945 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:46:29.950967 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 13:46:29.951042 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 13:46:29.951106 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5247,"thread_start_us":89,"threads_started":1}
I20260926 13:46:29.951356 24711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:46:29.940656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56338 (local address 127.12.170.65:39825)
0926 13:46:29.945851 (+ 5195us) server_negotiation.cc:206] Beginning negotiation
0926 13:46:29.945857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:46:29.945880 (+ 23us) server_negotiation.cc:407] Connection header received
0926 13:46:29.945950 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:46:29.945955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:46:29.946150 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:46:29.946291 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:46:29.946652 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:46:29.947996 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:46:29.949669 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:46:29.950515 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:46:29.950639 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:46:29.950784 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 13:46:29.951100 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 13:46:29.951201 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5081,"thread_start_us":87,"threads_started":1}
I20260926 13:46:29.953791 24707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:46:29.938580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60184 (local address 127.12.170.66:43457)
0926 13:46:29.941594 (+ 3014us) server_negotiation.cc:206] Beginning negotiation
0926 13:46:29.941606 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:46:29.941629 (+ 23us) server_negotiation.cc:407] Connection header received
0926 13:46:29.941716 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:46:29.941721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:46:29.941930 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:46:29.942076 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:46:29.942555 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:46:29.943762 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:46:29.945477 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:46:29.947059 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:46:29.948044 (+ 985us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:46:29.948291 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 13:46:29.948341 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 13:46:29.948431 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2909,"thread_start_us":146,"threads_started":1}
I20260926 13:46:29.954605 24356 tablet_service.cc:1467] Processing CreateTablet for tablet b20e6feb8e894af38eab571b091d153b (DEFAULT_TABLE table=test-table [id=15ddc506333149068a2d5365801e1134]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:46:29.956385 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:46:29.939370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:33533 (local address 127.0.0.1:34028)
0926 13:46:29.941267 (+ 1897us) negotiation.cc:107] Waiting for socket to connect
0926 13:46:29.941280 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:46:29.941356 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:46:29.952217 (+ 10861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:46:29.952220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:46:29.952237 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:46:29.952313 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:46:29.952319 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:46:29.953731 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:46:29.953734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:46:29.954882 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:46:29.954893 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:46:29.955998 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:46:29.956002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:46:29.956119 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:46:29.956144 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 13:46:29.956193 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 13:46:29.956256 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1836,"thread_start_us":87,"threads_started":1}
I20260926 13:46:29.956482 24494 tablet_service.cc:1467] Processing CreateTablet for tablet b20e6feb8e894af38eab571b091d153b (DEFAULT_TABLE table=test-table [id=15ddc506333149068a2d5365801e1134]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:46:29.956990 24709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:46:29.939765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34028 (local address 127.12.170.67:33533)
0926 13:46:29.951834 (+ 12069us) server_negotiation.cc:206] Beginning negotiation
0926 13:46:29.951842 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:46:29.951861 (+ 19us) server_negotiation.cc:407] Connection header received
0926 13:46:29.951921 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:46:29.951924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:46:29.952072 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:46:29.952224 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:46:29.952434 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:46:29.953587 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:46:29.955047 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:46:29.955868 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:46:29.956556 (+ 688us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:46:29.956711 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 13:46:29.956762 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 13:46:29.956848 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11975,"thread_start_us":120,"threads_started":1}
I20260926 13:46:29.957017 24356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b20e6feb8e894af38eab571b091d153b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:46:29.957499 24494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b20e6feb8e894af38eab571b091d153b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:46:29.959472 24628 tablet_service.cc:1467] Processing CreateTablet for tablet b20e6feb8e894af38eab571b091d153b (DEFAULT_TABLE table=test-table [id=15ddc506333149068a2d5365801e1134]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:46:29.960387 24628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b20e6feb8e894af38eab571b091d153b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:46:29.973649 24713 tablet_bootstrap.cc:492] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c: Bootstrap starting.
I20260926 13:46:29.975889 24712 tablet_bootstrap.cc:492] T b20e6feb8e894af38eab571b091d153b P f4cb1eac10c14954b295429379dfa711: Bootstrap starting.
I20260926 13:46:29.978489 24712 tablet_bootstrap.cc:654] T b20e6feb8e894af38eab571b091d153b P f4cb1eac10c14954b295429379dfa711: Neither blocks nor log segments found. Creating new log.
I20260926 13:46:29.979327 24712 log.cc:824] T b20e6feb8e894af38eab571b091d153b P f4cb1eac10c14954b295429379dfa711: Log is configured to *not* fsync() on all Append() calls
I20260926 13:46:29.979828 24713 tablet_bootstrap.cc:654] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c: Neither blocks nor log segments found. Creating new log.
I20260926 13:46:29.980014 24714 tablet_bootstrap.cc:492] T b20e6feb8e894af38eab571b091d153b P 71d5e7d42f1a485d895d865c04bf7263: Bootstrap starting.
I20260926 13:46:29.982301 24714 tablet_bootstrap.cc:654] T b20e6feb8e894af38eab571b091d153b P 71d5e7d42f1a485d895d865c04bf7263: Neither blocks nor log segments found. Creating new log.
I20260926 13:46:29.982997 24714 log.cc:824] T b20e6feb8e894af38eab571b091d153b P 71d5e7d42f1a485d895d865c04bf7263: Log is configured to *not* fsync() on all Append() calls
I20260926 13:46:29.984969 24712 tablet_bootstrap.cc:492] T b20e6feb8e894af38eab571b091d153b P f4cb1eac10c14954b295429379dfa711: No bootstrap required, opened a new log
I20260926 13:46:29.985157 24712 ts_tablet_manager.cc:1397] T b20e6feb8e894af38eab571b091d153b P f4cb1eac10c14954b295429379dfa711: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20260926 13:46:29.986753 24713 log.cc:824] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c: Log is configured to *not* fsync() on all Append() calls
I20260926 13:46:29.988785 24712 raft_consensus.cc:348] T b20e6feb8e894af38eab571b091d153b P f4cb1eac10c14954b295429379dfa711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cb1eac10c14954b295429379dfa711" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43457 } } peers { permanent_uuid: "71d5e7d42f1a485d895d865c04bf7263" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 33533 } } peers { permanent_uuid: "afcd7a87b05d474e9520461b091cb73c" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39825 } }
I20260926 13:46:29.989032 24712 raft_consensus.cc:374] T b20e6feb8e894af38eab571b091d153b P f4cb1eac10c14954b295429379dfa711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:46:29.989075 24712 raft_consensus.cc:729] T b20e6feb8e894af38eab571b091d153b P f4cb1eac10c14954b295429379dfa711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4cb1eac10c14954b295429379dfa711, State: Initialized, Role: FOLLOWER
I20260926 13:46:29.989528 24712 consensus_queue.cc:260] T b20e6feb8e894af38eab571b091d153b P f4cb1eac10c14954b295429379dfa711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cb1eac10c14954b295429379dfa711" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43457 } } peers { permanent_uuid: "71d5e7d42f1a485d895d865c04bf7263" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 33533 } } peers { permanent_uuid: "afcd7a87b05d474e9520461b091cb73c" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39825 } }
I20260926 13:46:29.990471 24712 ts_tablet_manager.cc:1428] T b20e6feb8e894af38eab571b091d153b P f4cb1eac10c14954b295429379dfa711: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260926 13:46:29.990795 24560 heartbeater.cc:503] Master 127.12.170.126:45155 was elected leader, sending a full tablet report...
I20260926 13:46:29.995241 24713 tablet_bootstrap.cc:492] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c: No bootstrap required, opened a new log
I20260926 13:46:29.995431 24713 ts_tablet_manager.cc:1397] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.002s
I20260926 13:46:29.996030 24714 tablet_bootstrap.cc:492] T b20e6feb8e894af38eab571b091d153b P 71d5e7d42f1a485d895d865c04bf7263: No bootstrap required, opened a new log
I20260926 13:46:29.996217 24714 ts_tablet_manager.cc:1397] T b20e6feb8e894af38eab571b091d153b P 71d5e7d42f1a485d895d865c04bf7263: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20260926 13:46:29.999606 24713 raft_consensus.cc:348] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cb1eac10c14954b295429379dfa711" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43457 } } peers { permanent_uuid: "71d5e7d42f1a485d895d865c04bf7263" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 33533 } } peers { permanent_uuid: "afcd7a87b05d474e9520461b091cb73c" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39825 } }
I20260926 13:46:29.999899 24713 raft_consensus.cc:374] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:46:29.999959 24713 raft_consensus.cc:729] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afcd7a87b05d474e9520461b091cb73c, State: Initialized, Role: FOLLOWER
I20260926 13:46:30.000007 24714 raft_consensus.cc:348] T b20e6feb8e894af38eab571b091d153b P 71d5e7d42f1a485d895d865c04bf7263 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cb1eac10c14954b295429379dfa711" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43457 } } peers { permanent_uuid: "71d5e7d42f1a485d895d865c04bf7263" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 33533 } } peers { permanent_uuid: "afcd7a87b05d474e9520461b091cb73c" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39825 } }
I20260926 13:46:30.000248 24714 raft_consensus.cc:374] T b20e6feb8e894af38eab571b091d153b P 71d5e7d42f1a485d895d865c04bf7263 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:46:30.000295 24714 raft_consensus.cc:729] T b20e6feb8e894af38eab571b091d153b P 71d5e7d42f1a485d895d865c04bf7263 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d5e7d42f1a485d895d865c04bf7263, State: Initialized, Role: FOLLOWER
I20260926 13:46:30.000463 24713 consensus_queue.cc:260] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cb1eac10c14954b295429379dfa711" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43457 } } peers { permanent_uuid: "71d5e7d42f1a485d895d865c04bf7263" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 33533 } } peers { permanent_uuid: "afcd7a87b05d474e9520461b091cb73c" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39825 } }
I20260926 13:46:30.000705 24714 consensus_queue.cc:260] T b20e6feb8e894af38eab571b091d153b P 71d5e7d42f1a485d895d865c04bf7263 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cb1eac10c14954b295429379dfa711" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43457 } } peers { permanent_uuid: "71d5e7d42f1a485d895d865c04bf7263" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 33533 } } peers { permanent_uuid: "afcd7a87b05d474e9520461b091cb73c" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39825 } }
I20260926 13:46:30.001483 24713 ts_tablet_manager.cc:1428] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260926 13:46:30.001549 24714 ts_tablet_manager.cc:1428] T b20e6feb8e894af38eab571b091d153b P 71d5e7d42f1a485d895d865c04bf7263: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 13:46:30.001816 24424 heartbeater.cc:503] Master 127.12.170.126:45155 was elected leader, sending a full tablet report...
I20260926 13:46:30.001843 24694 heartbeater.cc:503] Master 127.12.170.126:45155 was elected leader, sending a full tablet report...
W20260926 13:46:30.010360 24425 tablet.cc:2369] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 13:46:30 dist-test-slave-hc1k krb5kdc[23734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430389, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for HTTP/127.12.170.84@KRBTEST.COM
W20260926 13:46:30.074921 24561 tablet.cc:2369] T b20e6feb8e894af38eab571b091d153b P f4cb1eac10c14954b295429379dfa711: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:46:30.117980 24720 raft_consensus.cc:482] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:46:30.118234 24720 raft_consensus.cc:504] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cb1eac10c14954b295429379dfa711" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43457 } } peers { permanent_uuid: "71d5e7d42f1a485d895d865c04bf7263" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 33533 } } peers { permanent_uuid: "afcd7a87b05d474e9520461b091cb73c" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39825 } }
I20260926 13:46:30.119614 24720 leader_election.cc:288] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4cb1eac10c14954b295429379dfa711 (127.12.170.66:43457), 71d5e7d42f1a485d895d865c04bf7263 (127.12.170.67:33533)
I20260926 13:46:30.133930 24709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:46:30.120069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:52063 (local address 127.12.170.67:33533)
0926 13:46:30.120273 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 13:46:30.120278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:46:30.127916 (+ 7638us) server_negotiation.cc:407] Connection header received
0926 13:46:30.128119 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:46:30.128123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:46:30.128173 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:46:30.128221 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:46:30.128604 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:46:30.129285 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:46:30.132574 (+ 3289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:46:30.133194 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:46:30.133291 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:46:30.133348 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 13:46:30.133711 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 13:46:30.133794 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260926 13:46:30.134394 24724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:46:30.119914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:33533 (local address 127.12.170.65:52063)
0926 13:46:30.127822 (+ 7908us) negotiation.cc:107] Waiting for socket to connect
0926 13:46:30.127836 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:46:30.127989 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:46:30.128352 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:46:30.128355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:46:30.128369 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:46:30.128482 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:46:30.128488 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:46:30.129449 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:46:30.129453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:46:30.130563 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:46:30.130573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:46:30.133460 (+ 2887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:46:30.133463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:46:30.133561 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:46:30.133586 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 13:46:30.134161 (+ 575us) client_negotiation.cc:240] Negotiation successful
0926 13:46:30.134236 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7839,"mutex_wait_us":1,"thread_start_us":67,"threads_started":1}
I20260926 13:46:30.135185 24629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20e6feb8e894af38eab571b091d153b" candidate_uuid: "afcd7a87b05d474e9520461b091cb73c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d5e7d42f1a485d895d865c04bf7263" is_pre_election: true
I20260926 13:46:30.135499 24629 raft_consensus.cc:2454] T b20e6feb8e894af38eab571b091d153b P 71d5e7d42f1a485d895d865c04bf7263 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afcd7a87b05d474e9520461b091cb73c in term 0.
I20260926 13:46:30.136137 24314 leader_election.cc:302] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d5e7d42f1a485d895d865c04bf7263, afcd7a87b05d474e9520461b091cb73c; no voters:
I20260926 13:46:30.136469 24720 raft_consensus.cc:2781] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:46:30.136582 24720 raft_consensus.cc:482] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:46:30.136641 24720 raft_consensus.cc:3037] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:46:30.137926 24720 raft_consensus.cc:504] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cb1eac10c14954b295429379dfa711" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43457 } } peers { permanent_uuid: "71d5e7d42f1a485d895d865c04bf7263" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 33533 } } peers { permanent_uuid: "afcd7a87b05d474e9520461b091cb73c" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39825 } }
I20260926 13:46:30.138319 24720 leader_election.cc:288] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c [CANDIDATE]: Term 1 election: Requested vote from peers f4cb1eac10c14954b295429379dfa711 (127.12.170.66:43457), 71d5e7d42f1a485d895d865c04bf7263 (127.12.170.67:33533)
I20260926 13:46:30.138777 24629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20e6feb8e894af38eab571b091d153b" candidate_uuid: "afcd7a87b05d474e9520461b091cb73c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d5e7d42f1a485d895d865c04bf7263"
I20260926 13:46:30.138911 24629 raft_consensus.cc:3037] T b20e6feb8e894af38eab571b091d153b P 71d5e7d42f1a485d895d865c04bf7263 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:46:30.140324 24629 raft_consensus.cc:2454] T b20e6feb8e894af38eab571b091d153b P 71d5e7d42f1a485d895d865c04bf7263 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afcd7a87b05d474e9520461b091cb73c in term 1.
W20260926 13:46:30.140395 24695 tablet.cc:2369] T b20e6feb8e894af38eab571b091d153b P 71d5e7d42f1a485d895d865c04bf7263: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:46:30.140740 24314 leader_election.cc:302] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d5e7d42f1a485d895d865c04bf7263, afcd7a87b05d474e9520461b091cb73c; no voters:
I20260926 13:46:30.140900 24720 raft_consensus.cc:2781] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:46:30.141085 24707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:46:30.120365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:50849 (local address 127.12.170.66:43457)
0926 13:46:30.120470 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 13:46:30.120474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:46:30.127475 (+ 7001us) server_negotiation.cc:407] Connection header received
0926 13:46:30.127671 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:46:30.127674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:46:30.127733 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:46:30.127819 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:46:30.128262 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:46:30.129003 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:46:30.139771 (+ 10768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:46:30.140408 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:46:30.140504 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:46:30.140554 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:46:30.140885 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 13:46:30.140970 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:46:30.141129 24720 raft_consensus.cc:686] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c [term 1 LEADER]: Becoming Leader. State: Replica: afcd7a87b05d474e9520461b091cb73c, State: Running, Role: LEADER
I20260926 13:46:30.141492 24723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:46:30.119824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:43457 (local address 127.12.170.65:50849)
0926 13:46:30.127349 (+ 7525us) negotiation.cc:107] Waiting for socket to connect
0926 13:46:30.127367 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:46:30.127528 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:46:30.127939 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:46:30.127942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:46:30.127961 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:46:30.128105 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:46:30.128113 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:46:30.129148 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:46:30.129151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:46:30.130396 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:46:30.130406 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:46:30.140645 (+ 10239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:46:30.140649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:46:30.140755 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:46:30.140775 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 13:46:30.141274 (+ 499us) client_negotiation.cc:240] Negotiation successful
0926 13:46:30.141353 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7423,"thread_start_us":138,"threads_started":1}
I20260926 13:46:30.141480 24720 consensus_queue.cc:237] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cb1eac10c14954b295429379dfa711" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43457 } } peers { permanent_uuid: "71d5e7d42f1a485d895d865c04bf7263" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 33533 } } peers { permanent_uuid: "afcd7a87b05d474e9520461b091cb73c" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39825 } }
I20260926 13:46:30.142335 24514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20e6feb8e894af38eab571b091d153b" candidate_uuid: "afcd7a87b05d474e9520461b091cb73c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4cb1eac10c14954b295429379dfa711" is_pre_election: true
I20260926 13:46:30.142349 24513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20e6feb8e894af38eab571b091d153b" candidate_uuid: "afcd7a87b05d474e9520461b091cb73c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4cb1eac10c14954b295429379dfa711"
I20260926 13:46:30.142606 24514 raft_consensus.cc:2454] T b20e6feb8e894af38eab571b091d153b P f4cb1eac10c14954b295429379dfa711 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afcd7a87b05d474e9520461b091cb73c in term 0.
I20260926 13:46:30.144987 24210 catalog_manager.cc:5495] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c reported cstate change: term changed from 0 to 1, leader changed from <none> to afcd7a87b05d474e9520461b091cb73c (127.12.170.65). New cstate: current_term: 1 leader_uuid: "afcd7a87b05d474e9520461b091cb73c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4cb1eac10c14954b295429379dfa711" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71d5e7d42f1a485d895d865c04bf7263" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 33533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afcd7a87b05d474e9520461b091cb73c" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39825 } health_report { overall_health: HEALTHY } } }
I20260926 13:46:30.184521 24711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:46:30.179770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56344 (local address 127.12.170.65:39825)
0926 13:46:30.179908 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 13:46:30.179913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:46:30.179933 (+ 20us) server_negotiation.cc:407] Connection header received
0926 13:46:30.180021 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:46:30.180024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:46:30.180092 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:46:30.180209 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:46:30.180471 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:46:30.181275 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:46:30.182485 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:46:30.182832 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:46:30.182938 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:46:30.183290 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:46:30.183865 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:46:30.183974 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 13:46:30.184332 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 13:46:30.184388 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 13:46:30.191102 24513 raft_consensus.cc:3037] T b20e6feb8e894af38eab571b091d153b P f4cb1eac10c14954b295429379dfa711 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:46:30.192919 24513 raft_consensus.cc:1261] T b20e6feb8e894af38eab571b091d153b P f4cb1eac10c14954b295429379dfa711 [term 1 FOLLOWER]: Refusing update from remote peer afcd7a87b05d474e9520461b091cb73c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:46:30.193805 24725 consensus_queue.cc:1035] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4cb1eac10c14954b295429379dfa711" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 43457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:46:30.197124 24629 raft_consensus.cc:1261] T b20e6feb8e894af38eab571b091d153b P 71d5e7d42f1a485d895d865c04bf7263 [term 1 FOLLOWER]: Refusing update from remote peer afcd7a87b05d474e9520461b091cb73c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:46:30.204384 24725 consensus_queue.cc:1035] T b20e6feb8e894af38eab571b091d153b P afcd7a87b05d474e9520461b091cb73c [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d5e7d42f1a485d895d865c04bf7263" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 33533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:46:30.212203 24729 mvcc.cc:205] Tried to move back new op lower bound from 7333602878215299072 to 7333602878022307840. Current Snapshot: MvccSnapshot[applied={T|T < 7333602878215299072}]
I20260926 13:46:30.222913 24209 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:46:30.223105 24209 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:46:30.225692 24209 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a7fe52d5f1d4396808a1e012cd5a2b7: Sending DeleteTablet for 3 replicas of tablet b20e6feb8e894af38eab571b091d153b
I20260926 13:46:30.226841 24628 tablet_service.cc:1514] Processing DeleteTablet for tablet b20e6feb8e894af38eab571b091d153b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:46:30 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:34028
I20260926 13:46:30.226846 24494 tablet_service.cc:1514] Processing DeleteTablet for tablet b20e6feb8e894af38eab571b091d153b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:46:30 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:60184
I20260926 13:46:30.227633 24356 tablet_service.cc:1514] Processing DeleteTablet for tablet b20e6feb8e894af38eab571b091d153b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:46:30 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:56338
I20260926 13:46:30.228333 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 24291
I20260926 13:46:30.229779 24738 tablet_replica.cc:330] T b20e6feb8e894af38eab571b091d153b P f4cb1eac10c14954b295429379dfa711: stopping tablet replica
I20260926 13:46:30.230057 24738 raft_consensus.cc:2229] T b20e6feb8e894af38eab571b091d153b P f4cb1eac10c14954b295429379dfa711 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:46:30.230335 24738 raft_consensus.cc:2258] T b20e6feb8e894af38eab571b091d153b P f4cb1eac10c14954b295429379dfa711 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:46:30.231161 24737 tablet_replica.cc:330] T b20e6feb8e894af38eab571b091d153b P 71d5e7d42f1a485d895d865c04bf7263: stopping tablet replica
I20260926 13:46:30.231410 24737 raft_consensus.cc:2229] T b20e6feb8e894af38eab571b091d153b P 71d5e7d42f1a485d895d865c04bf7263 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:46:30.231693 24737 raft_consensus.cc:2258] T b20e6feb8e894af38eab571b091d153b P 71d5e7d42f1a485d895d865c04bf7263 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:46:30.233244 24737 ts_tablet_manager.cc:1905] T b20e6feb8e894af38eab571b091d153b P 71d5e7d42f1a485d895d865c04bf7263: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:46:30.235862 24737 ts_tablet_manager.cc:1918] T b20e6feb8e894af38eab571b091d153b P 71d5e7d42f1a485d895d865c04bf7263: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:46:30.235942 24737 log.cc:1196] T b20e6feb8e894af38eab571b091d153b P 71d5e7d42f1a485d895d865c04bf7263: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/b20e6feb8e894af38eab571b091d153b
I20260926 13:46:30.236246 24737 ts_tablet_manager.cc:1939] T b20e6feb8e894af38eab571b091d153b P 71d5e7d42f1a485d895d865c04bf7263: Deleting consensus metadata
I20260926 13:46:30.237424 24197 catalog_manager.cc:4841] TS 71d5e7d42f1a485d895d865c04bf7263 (127.12.170.67:33533): tablet b20e6feb8e894af38eab571b091d153b (table test-table [id=15ddc506333149068a2d5365801e1134]) successfully deleted
I20260926 13:46:30.238539 24738 ts_tablet_manager.cc:1905] T b20e6feb8e894af38eab571b091d153b P f4cb1eac10c14954b295429379dfa711: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 13:46:30.238958 24194 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.170.65:39825 (error 108)
I20260926 13:46:30.239212 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 24430
I20260926 13:46:30.240641 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:46:30.240373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:39825 (local address 127.0.0.1:56354)
0926 13:46:30.240501 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 13:46:30.240567 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.65:39825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 13:46:30.240797 24194 catalog_manager.cc:4584] TS afcd7a87b05d474e9520461b091cb73c (127.12.170.65:39825): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b20e6feb8e894af38eab571b091d153b: Network error: Client connection negotiation failed: client connection to 127.12.170.65:39825: connect: Connection refused (error 111)
I20260926 13:46:30.245433 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:46:30.245149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:43457 (local address 127.0.0.1:60198)
0926 13:46:30.245313 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 13:46:30.245361 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.66:43457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 13:46:30.246479 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 24565
I20260926 13:46:30.251772 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 24178
I20260926 13:46:30.256517 12969 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 13:46:30.590356 12969 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 13:46:30.590480 12969 mini_ranger.cc:66] Stopping Ranger...
I20260926 13:46:30.924471 12969 mini_ranger.cc:68] Stopped Ranger
2026-09-26 13:46:30.924 UTC [23764] LOG: received smart shutdown request
2026-09-26 13:46:30.927 UTC [23764] LOG: background worker "logical replication launcher" (PID 23773) exited with exit code 1
2026-09-26 13:46:30.927 UTC [23768] LOG: shutting down
2026-09-26 13:46:31.073 UTC [23764] LOG: database system is shut down
2026-09-26T13:46:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (42297 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:46:31 dist-test-slave-hc1k krb5kdc[24749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:46:31 dist-test-slave-hc1k krb5kdc[24749](info): set up 2 sockets
Sep 26 13:46:31 dist-test-slave-hc1k krb5kdc[24749](info): commencing operation
krb5kdc: starting...
W20260926 13:46:32.111425 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:46:32 dist-test-slave-hc1k krb5kdc[24749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:46:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:46:32Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.170.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.170.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.170.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/rangeradmin_127.12.170.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.170.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/rangeradmin_127.12.170.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.170.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.170.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.170.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/rangerlookup_127.12.170.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.170.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/rangerlookup_127.12.170.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.170.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.170.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.170.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/HTTP_127.12.170.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.170.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/HTTP_127.12.170.84@KRBTEST.COM.keytab.
I20260926 13:46:32.261732 12969 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task0bSs4M/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/postgres -l logfile start
2026-09-26 13:46:34.524 UTC [24779] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 13:46:34.524 UTC [24779] LOG: listening on IPv4 address "127.12.170.84", port 43821
2026-09-26 13:46:34.528 UTC [24779] LOG: listening on Unix socket "/tmp/.s.PGSQL.43821"
2026-09-26 13:46:34.547 UTC [24782] LOG: database system was shut down at 2026-09-26 13:46:33 UTC
2026-09-26 13:46:34.550 UTC [24779] LOG: database system is ready to accept connections
I20260926 13:46:35.419314 12969 mini_postgres.cc:96] Postgres bound to 43821
2026-09-26 13:46:35.425 UTC [24790] FATAL: database "slave" does not exist
127.12.170.84:43821 - accepting connections
I20260926 13:46:35.425549 12969 mini_ranger.cc:161] Starting Ranger...
I20260926 13:46:35.438943 12969 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 13:46:35.860939 12969 mini_ranger.cc:87] Created ranger Postgres database
I20260926 13:46:35.861011 12969 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ranger-admin
2026-09-26 13:46:36,122 [I] DB FLAVOR :POSTGRES
2026-09-26 13:46:36,122 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 13:46:36,122 [I] Checking connection..
2026-09-26 13:46:36,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:43821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 13:46:36,355 [I] Checking connection passed.
2026-09-26 13:46:36,356 [I] --------- Verifying version history table ---------
2026-09-26 13:46:36,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:43821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 13:46:36,602 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 13:46:36,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:43821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 13:46:36,847 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 13:46:36,847 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 13:46:36,847 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:43821/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 13:46:37.064 UTC [24854] WARNING: there is no transaction in progress
2026-09-26 13:46:37,069 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 13:46:37,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:43821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 13:46:37,303 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 13:46:37,303 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 13:46:37,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:43821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 13:46:37,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:43821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 13:46:37,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:43821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hc1k.c.gcp-upstream.internal','N') ;"
2026-09-26 13:46:37,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:43821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 13:46:38,240 [I] Table x_portal_user does not exist in database ranger
2026-09-26 13:46:38,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:43821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 13:46:38,472 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 13:46:38,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:43821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 13:46:38,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:43821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 13:46:38,961 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 13:46:38,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:43821/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 13:46:39.839 UTC [24962] WARNING: there is no transaction in progress
2026-09-26 13:46:39.854 UTC [24962] WARNING: there is no transaction in progress
2026-09-26 13:46:39.869 UTC [24962] WARNING: there is no transaction in progress
2026-09-26 13:46:39.883 UTC [24962] WARNING: there is no transaction in progress
2026-09-26 13:46:39.901 UTC [24962] WARNING: there is no transaction in progress
2026-09-26 13:46:39.919 UTC [24962] WARNING: there is no transaction in progress
2026-09-26 13:46:39.971 UTC [24962] WARNING: there is no transaction in progress
2026-09-26 13:46:39.978 UTC [24962] WARNING: there is no transaction in progress
2026-09-26 13:46:39.991 UTC [24962] WARNING: there is no transaction in progress
2026-09-26 13:46:40.000 UTC [24962] WARNING: there is no transaction in progress
2026-09-26 13:46:40.010 UTC [24962] WARNING: there is no transaction in progress
2026-09-26 13:46:40.023 UTC [24962] WARNING: there is no transaction in progress
2026-09-26 13:46:40.031 UTC [24962] WARNING: there is no transaction in progress
2026-09-26 13:46:40.037 UTC [24962] WARNING: there is no transaction in progress
2026-09-26 13:46:40.043 UTC [24962] WARNING: there is no transaction in progress
2026-09-26 13:46:40.050 UTC [24962] WARNING: there is no transaction in progress
2026-09-26 13:46:40,646 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 13:46:40,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:43821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 13:46:40,867 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 13:46:40,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:43821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 13:46:41,090 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 13:46:41,090 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:43821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 13:46:41,327 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 13:46:41,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:43821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 13:46:41,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:43821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 13:46:41,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:43821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hc1k.c.gcp-upstream.internal';"
2026-09-26 13:46:42,031 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 13:46:42,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:43821/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 13:46:42,275 [I] DB_PATCHES have already been applied
I20260926 13:46:42.280334 12969 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ranger-admin:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task0bSs4M/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 13:46:42.280393 12969 mini_ranger.cc:193] Using host: 127.12.170.84
I20260926 13:46:42.283641 12969 mini_ranger.cc:227] Ranger admin URL: http://127.12.170.84:53551
Sep 26, 2026 1:46:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:46:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:46:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 1:46:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 1:46:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 1:46:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 1:46:43 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 13:46:43.273764 12969 mini_ranger.cc:160] Time spent starting Ranger: real 7.848s user 0.000s sys 0.008s
Sep 26, 2026 1:47:03 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 1:47:03 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 1:47:03 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 1:47:03 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 1:47:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 1:47:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 1:47:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 1:47:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 1:47:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 1:47:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 1:47:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 1:47:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 1:47:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 1:47:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 1:47:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 1:47:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 1:47:03 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 1:47:04 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 13:47:04.562196 12969 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.170.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.170.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.170.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/rangerkms_127.12.170.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.170.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/rangerkms_127.12.170.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.170.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.170.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.170.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/HTTP_127.12.170.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.170.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/HTTP_127.12.170.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 13:47:04 dist-test-slave-hc1k krb5kdc[24749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430424, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 13:47:04.658195 12969 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 13:47:04.666684 12969 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 13:47:05.457549 12969 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 13:47:05.457618 12969 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ranger-kms
I20260926 13:47:05.457636 12969 mini_ranger_kms.cc:227] Using postgres at 127.12.170.84:43821
2026-09-26 13:47:05,607 [I] DB FLAVOR :POSTGRES
2026-09-26 13:47:05,607 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 13:47:05,607 [I] Checking connection
2026-09-26 13:47:05,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:43821/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 13:47:05,852 [I] connection success
2026-09-26 13:47:05,853 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 13:47:05,853 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 13:47:05,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:43821/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 13:47:06,072 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 13:47:06,073 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 13:47:06,073 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 13:47:06,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.170.84:43821/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 13:47:06,325 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 13:47:06.807682 12969 mini_ranger_kms.cc:310] Created kms service
I20260926 13:47:06.983566 12969 mini_ranger_kms.cc:326] Created kudu user
I20260926 13:47:07.025755 12969 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 13:47:07 dist-test-slave-hc1k krb5kdc[24749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430424, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.170.84@KRBTEST.COM
I20260926 13:47:07.393584 12969 mini_ranger_kms.cc:384] Added ranger policy
I20260926 13:47:07.393812 12969 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ranger-kms:/tmp/dist-test-task0bSs4M/build/debug/bin/postgresql.jar:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task0bSs4M/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 13:47:07.393841 12969 mini_ranger_kms.cc:240] Using host: 127.12.170.84
I20260926 13:47:07.397167 12969 mini_ranger_kms.cc:276] Ranger KMS PID: 25142
I20260926 13:47:07.397239 12969 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.170.84:38147
Sep 26, 2026 1:47:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:47:08 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:47:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 1:47:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 1:47:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 1:47:08 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 13:47:09.402951 12969 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.745s user 0.008s sys 0.125s
I20260926 13:47:09.403087 12969 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 13:47:09.403169 12969 mini_ranger_kms.cc:401] 127.12.170.84:38147/kms/v1/keys
Sep 26 13:47:09 dist-test-slave-hc1k krb5kdc[24749](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790430429, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.170.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 13:47:10 dist-test-slave-hc1k krb5kdc[24749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430429, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.170.84@KRBTEST.COM for HTTP/127.12.170.84@KRBTEST.COM
Sep 26 13:47:10 dist-test-slave-hc1k krb5kdc[24749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430429, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.170.84@KRBTEST.COM for HTTP/127.12.170.84@KRBTEST.COM
Sep 26 13:47:12 dist-test-slave-hc1k krb5kdc[24749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:47:12.480333 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:33809
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:46113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:33809
--ranger_config_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.170.84:38147/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:47:12.585258 25193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:12.585491 25193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:12.585525 25193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:12.589418 25193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:47:12.589475 25193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:12.589493 25193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:12.589509 25193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:47:12.589535 25193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:47:12.594630 25193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:46113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.170.84:38147/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:33809
--ranger_config_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:33809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:12.595546 25193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:12.596491 25193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:47:12.604619 25199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:12.604718 25193 server_base.cc:1034] running on GCE node
W20260926 13:47:12.604628 25198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:12.605031 25201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:12.605552 25193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:12.606599 25193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:12.607712 25193 hybrid_clock.cc:648] HybridClock initialized: now 1790430432607687 us; error 36 us; skew 500 ppm
Sep 26 13:47:12 dist-test-slave-hc1k krb5kdc[24749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430432, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:12.610181 25193 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:47:12.610996 25193 webserver.cc:460] Webserver started at http://127.12.170.126:42985/ using document root <none> and password file <none>
I20260926 13:47:12.611474 25193 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:12.611532 25193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:47:12.611740 25193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:47:12 dist-test-slave-hc1k krb5kdc[24749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430432, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for HTTP/127.12.170.84@KRBTEST.COM
I20260926 13:47:12.657032 25193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "dceb4d13591a4cb2961bd52aa628cf7f"
format_stamp: "Formatted at 2026-09-26 13:47:12 on dist-test-slave-hc1k"
server_key: "98ce42c7d1a8fde0732aec15a0a34f3c"
server_key_iv: "b8596ac74bc4b59f2cf98e6623c34434"
server_key_version: "kuduclusterkey@0"
I20260926 13:47:12.657617 25193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "dceb4d13591a4cb2961bd52aa628cf7f"
format_stamp: "Formatted at 2026-09-26 13:47:12 on dist-test-slave-hc1k"
server_key: "98ce42c7d1a8fde0732aec15a0a34f3c"
server_key_iv: "b8596ac74bc4b59f2cf98e6623c34434"
server_key_version: "kuduclusterkey@0"
I20260926 13:47:12.661203 25193 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260926 13:47:12.676538 25208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:12.677798 25193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:47:12.677914 25193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "dceb4d13591a4cb2961bd52aa628cf7f"
format_stamp: "Formatted at 2026-09-26 13:47:12 on dist-test-slave-hc1k"
server_key: "98ce42c7d1a8fde0732aec15a0a34f3c"
server_key_iv: "b8596ac74bc4b59f2cf98e6623c34434"
server_key_version: "kuduclusterkey@0"
I20260926 13:47:12.678033 25193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:12.691274 25193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:12.693832 25193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:12.693985 25193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:12.701047 25193 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:33809
I20260926 13:47:12.701031 25260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:33809 every 8 connection(s)
I20260926 13:47:12.701915 25193 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:47:12.704469 25261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:47:12.706491 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 25193
I20260926 13:47:12.706598 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:47:12.706866 12969 external_mini_cluster.cc:1428] Setting key b2e468edfb82d7ca5900c63f8a896516
I20260926 13:47:12.710328 25261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f: Bootstrap starting.
I20260926 13:47:12.712764 25261 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f: Neither blocks nor log segments found. Creating new log.
I20260926 13:47:12.713416 25261 log.cc:824] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f: Log is configured to *not* fsync() on all Append() calls
I20260926 13:47:12.715288 25261 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f: No bootstrap required, opened a new log
Sep 26 13:47:12 dist-test-slave-hc1k krb5kdc[24749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:47:12.719190 25261 raft_consensus.cc:348] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dceb4d13591a4cb2961bd52aa628cf7f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33809 } }
I20260926 13:47:12.719424 25261 raft_consensus.cc:374] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:47:12.719528 25261 raft_consensus.cc:729] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dceb4d13591a4cb2961bd52aa628cf7f, State: Initialized, Role: FOLLOWER
I20260926 13:47:12.720070 25261 consensus_queue.cc:260] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dceb4d13591a4cb2961bd52aa628cf7f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33809 } }
I20260926 13:47:12.720218 25261 raft_consensus.cc:388] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:47:12.720316 25261 raft_consensus.cc:482] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:47:12.720434 25261 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:47:12.721400 25261 raft_consensus.cc:504] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dceb4d13591a4cb2961bd52aa628cf7f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33809 } }
I20260926 13:47:12.721748 25261 leader_election.cc:302] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dceb4d13591a4cb2961bd52aa628cf7f; no voters:
I20260926 13:47:12.721997 25264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:12.708200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40810 (local address 127.12.170.126:33809)
0926 13:47:12.708609 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:12.708627 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:12.708667 (+ 40us) server_negotiation.cc:407] Connection header received
0926 13:47:12.709411 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:12.709432 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:12.709806 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:12.710139 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:12.710931 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:12.712345 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:12.713496 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:12.714091 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:12.714238 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:12.717048 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:12.717054 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:12.717069 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:12.717098 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:12.718872 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:12.719328 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:12.719334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:12.719340 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:12.719464 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:12.719823 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:12.719826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:12.719827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:12.720294 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:12.720492 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:12.720747 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 13:47:12.721146 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":273,"thread_start_us":122,"threads_started":1}
I20260926 13:47:12.722141 25266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:47:12.722054 25261 leader_election.cc:288] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:47:12.723223 25266 raft_consensus.cc:686] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [term 1 LEADER]: Becoming Leader. State: Replica: dceb4d13591a4cb2961bd52aa628cf7f, State: Running, Role: LEADER
I20260926 13:47:12.723342 25261 sys_catalog.cc:564] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:47:12.723569 25266 consensus_queue.cc:237] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dceb4d13591a4cb2961bd52aa628cf7f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33809 } }
I20260926 13:47:12.725667 25267 sys_catalog.cc:455] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dceb4d13591a4cb2961bd52aa628cf7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dceb4d13591a4cb2961bd52aa628cf7f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33809 } } }
I20260926 13:47:12.725714 25261 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 13:47:12.725790 25267 sys_catalog.cc:458] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [sys.catalog]: This master's current role is: LEADER
I20260926 13:47:12.726557 25268 sys_catalog.cc:455] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader dceb4d13591a4cb2961bd52aa628cf7f. Latest consensus state: current_term: 1 leader_uuid: "dceb4d13591a4cb2961bd52aa628cf7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dceb4d13591a4cb2961bd52aa628cf7f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33809 } } }
I20260926 13:47:12.726723 25268 sys_catalog.cc:458] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [sys.catalog]: This master's current role is: LEADER
I20260926 13:47:12.726935 25280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:47:12.732457 25280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:47:12.741307 25280 catalog_manager.cc:1347] Generated new cluster ID: a720acb978504118afdec5bef537fda8
I20260926 13:47:12.741611 25280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:47:12.753747 25280 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:47:12.754922 25280 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:47:12.761920 25280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f: Generated new TSK 0
I20260926 13:47:12.762674 25280 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.170.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
I20260926 13:47:13.575182 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.170.84:38147/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:0
--local_ip_for_outbound_sockets=127.12.170.65
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:33809
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:46113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:47:13.709679 25306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:13.710029 25306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:13.710122 25306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:13.715931 25306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:13.716046 25306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:13.716163 25306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:47:13.723680 25306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:46113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.170.84:38147/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:33809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:13.725154 25306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:13.726406 25306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:47:13.735669 25312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:13.735764 25311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:13.736217 25314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:13.736717 25306 server_base.cc:1034] running on GCE node
I20260926 13:47:13.737372 25306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:13.738063 25306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:13.739239 25306 hybrid_clock.cc:648] HybridClock initialized: now 1790430433739221 us; error 38 us; skew 500 ppm
Sep 26 13:47:13 dist-test-slave-hc1k krb5kdc[24749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430433, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:13.742480 25306 init.cc:377] Logged in from keytab as kudu/127.12.170.65@KRBTEST.COM (short username kudu)
I20260926 13:47:13.743428 25306 webserver.cc:460] Webserver started at http://127.12.170.65:34539/ using document root <none> and password file <none>
I20260926 13:47:13.744079 25306 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:13.744182 25306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:47:13.744400 25306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:47:13 dist-test-slave-hc1k krb5kdc[24749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430433, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for HTTP/127.12.170.84@KRBTEST.COM
I20260926 13:47:13.759335 25306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/data/instance:
uuid: "182083588c30417e8a1b7c8360e17248"
format_stamp: "Formatted at 2026-09-26 13:47:13 on dist-test-slave-hc1k"
server_key: "0adf18b1a3a945e9da0a83e264d73514"
server_key_iv: "00bda0c08a77394a16d478f08e7a162f"
server_key_version: "kuduclusterkey@0"
I20260926 13:47:13.760006 25306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance:
uuid: "182083588c30417e8a1b7c8360e17248"
format_stamp: "Formatted at 2026-09-26 13:47:13 on dist-test-slave-hc1k"
server_key: "0adf18b1a3a945e9da0a83e264d73514"
server_key_iv: "00bda0c08a77394a16d478f08e7a162f"
server_key_version: "kuduclusterkey@0"
I20260926 13:47:13.764595 25306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:47:13.780464 25323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:13.782152 25306 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 13:47:13.782285 25306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "182083588c30417e8a1b7c8360e17248"
format_stamp: "Formatted at 2026-09-26 13:47:13 on dist-test-slave-hc1k"
server_key: "0adf18b1a3a945e9da0a83e264d73514"
server_key_iv: "00bda0c08a77394a16d478f08e7a162f"
server_key_version: "kuduclusterkey@0"
I20260926 13:47:13.782428 25306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:13.803584 25306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:13.806275 25306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:13.806435 25306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:13.807008 25306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:47:13.807868 25306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:47:13.807916 25306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:13.807969 25306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:47:13.808010 25306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:13.817191 25306 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:44653
I20260926 13:47:13.818250 25306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:47:13.822453 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 25306
I20260926 13:47:13.822552 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance
I20260926 13:47:13.822831 12969 external_mini_cluster.cc:1428] Setting key 20f5329b89836fc3f020a9c84efd1f3e
Sep 26 13:47:13 dist-test-slave-hc1k krb5kdc[24749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430433, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:47:13.834808 25439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:13.820987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:58925 (local address 127.12.170.126:33809)
0926 13:47:13.821231 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:13.821235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:13.823141 (+ 1906us) server_negotiation.cc:407] Connection header received
0926 13:47:13.823196 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:13.823199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:13.823304 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:13.823388 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:13.824953 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:13.825744 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:13.827380 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:13.827783 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:13.827860 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:13.830629 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:13.830632 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:13.830635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:13.830658 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:13.832434 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:13.833042 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:13.833046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:13.833047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:13.833136 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:13.833540 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:13.833543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:13.833545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:13.833755 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:13.833823 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:13.834489 (+ 666us) server_negotiation.cc:299] Negotiation successful
0926 13:47:13.834621 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":82,"threads_started":1}
I20260926 13:47:13.836581 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:13.821404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33809 (local address 127.12.170.65:58925)
0926 13:47:13.821842 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:13.821892 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:13.822859 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:13.823742 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:13.823751 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:13.824072 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:13.824792 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:13.824812 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:13.825888 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:13.825891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:13.827234 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:13.827246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:13.828046 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:13.828049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:13.828250 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:13.828827 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:13.828847 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:13.830459 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:13.832585 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:13.832591 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:13.832605 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:13.832924 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:13.833243 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:13.833246 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:13.833248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:13.833414 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:13.833903 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:13.833910 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:13.834200 (+ 290us) client_negotiation.cc:769] Sending connection context
0926 13:47:13.835235 (+ 1035us) client_negotiation.cc:240] Negotiation successful
0926 13:47:13.835567 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":113,"threads_started":1}
I20260926 13:47:13.838335 25437 heartbeater.cc:347] Connected to a master server at 127.12.170.126:33809
I20260926 13:47:13.838559 25437 heartbeater.cc:464] Registering TS with master...
I20260926 13:47:13.839069 25437 heartbeater.cc:511] Master 127.12.170.126:33809 requested a full tablet report, sending...
I20260926 13:47:13.840873 25225 ts_manager.cc:194] Registered new tserver with Master: 182083588c30417e8a1b7c8360e17248 (127.12.170.65:44653)
I20260926 13:47:13.841252 25436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:44653 every 8 connection(s)
I20260926 13:47:13.842397 25225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.65@KRBTEST.COM'} at 127.12.170.65:58925
WARNING: no policy specified for kudu/127.12.170.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
I20260926 13:47:13.878780 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.170.84:38147/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:0
--local_ip_for_outbound_sockets=127.12.170.66
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:33809
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:46113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:47:13 dist-test-slave-hc1k krb5kdc[24749](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790430433, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 13:47:14.045488 25445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:14.045890 25445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:14.045969 25445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:14.051879 25445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:14.052011 25445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:14.052131 25445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:47:14.059988 25445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:46113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.170.84:38147/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:33809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:14.061422 25445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:14.062831 25445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:47:14.075066 25454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:14.075263 25445 server_base.cc:1034] running on GCE node
W20260926 13:47:14.075124 25451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:14.075141 25452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:14.076256 25445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:14.077000 25445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:14.078187 25445 hybrid_clock.cc:648] HybridClock initialized: now 1790430434078158 us; error 45 us; skew 500 ppm
Sep 26 13:47:14 dist-test-slave-hc1k krb5kdc[24749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430434, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:14.081481 25445 init.cc:377] Logged in from keytab as kudu/127.12.170.66@KRBTEST.COM (short username kudu)
I20260926 13:47:14.082405 25445 webserver.cc:460] Webserver started at http://127.12.170.66:45877/ using document root <none> and password file <none>
I20260926 13:47:14.083010 25445 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:14.083097 25445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:47:14.083386 25445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:47:14 dist-test-slave-hc1k krb5kdc[24749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430434, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for HTTP/127.12.170.84@KRBTEST.COM
I20260926 13:47:14.101867 25445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/data/instance:
uuid: "9d067b92ac2d4b9f8ea3ffe2f51c8d9c"
format_stamp: "Formatted at 2026-09-26 13:47:14 on dist-test-slave-hc1k"
server_key: "4af6c01b82cd9be34b0e18be5a8fb5a4"
server_key_iv: "ba7ce1cc2e64ce37382d12af201a533e"
server_key_version: "kuduclusterkey@0"
I20260926 13:47:14.102689 25445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance:
uuid: "9d067b92ac2d4b9f8ea3ffe2f51c8d9c"
format_stamp: "Formatted at 2026-09-26 13:47:14 on dist-test-slave-hc1k"
server_key: "4af6c01b82cd9be34b0e18be5a8fb5a4"
server_key_iv: "ba7ce1cc2e64ce37382d12af201a533e"
server_key_version: "kuduclusterkey@0"
I20260926 13:47:14.108163 25445 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 13:47:14.132027 25461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:14.133875 25445 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.000s
I20260926 13:47:14.134022 25445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "9d067b92ac2d4b9f8ea3ffe2f51c8d9c"
format_stamp: "Formatted at 2026-09-26 13:47:14 on dist-test-slave-hc1k"
server_key: "4af6c01b82cd9be34b0e18be5a8fb5a4"
server_key_iv: "ba7ce1cc2e64ce37382d12af201a533e"
server_key_version: "kuduclusterkey@0"
I20260926 13:47:14.134156 25445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:14.154085 25445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:14.157725 25445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:14.157970 25445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:14.158761 25445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:47:14.159857 25445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:47:14.159973 25445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:14.160085 25445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:47:14.160148 25445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:14.172921 25445 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:45035
I20260926 13:47:14.174134 25445 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:47:14.174914 25574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:45035 every 8 connection(s)
I20260926 13:47:14.181416 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 25445
I20260926 13:47:14.181530 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance
I20260926 13:47:14.181820 12969 external_mini_cluster.cc:1428] Setting key 60dcea31a8e7b1c96124329470a59f8e
Sep 26 13:47:14 dist-test-slave-hc1k krb5kdc[24749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430434, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.170.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.67@KRBTEST.COM" created.
I20260926 13:47:14.202128 25439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:14.178919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:40021 (local address 127.12.170.126:33809)
0926 13:47:14.179053 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:14.179058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:14.187089 (+ 8031us) server_negotiation.cc:407] Connection header received
0926 13:47:14.188426 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:14.188429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:14.188485 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:14.188575 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:14.190256 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:14.191084 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:14.192793 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:14.193165 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:14.193245 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:14.197225 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:14.197229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:14.197267 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:14.197296 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:14.199639 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:14.200294 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:14.200298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:14.200300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:14.200388 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:14.200784 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:14.200788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:14.200790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:14.201023 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:14.201104 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:14.201784 (+ 680us) server_negotiation.cc:299] Negotiation successful
0926 13:47:14.201918 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:47:14.203768 25577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:14.177862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33809 (local address 127.12.170.66:40021)
0926 13:47:14.186905 (+ 9043us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:14.186977 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:14.188171 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:14.188977 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:14.188988 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:14.189371 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:14.190078 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:14.190100 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:14.191243 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:14.191247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:14.192658 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:14.192669 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:14.193446 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:14.193450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:14.193645 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:14.194333 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:14.194354 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:14.196915 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:14.199811 (+ 2896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:14.199818 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:14.199835 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:14.200177 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:14.200492 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:14.200495 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:14.200497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:14.200676 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:14.201171 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:14.201177 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:14.201505 (+ 328us) client_negotiation.cc:769] Sending connection context
0926 13:47:14.202364 (+ 859us) client_negotiation.cc:240] Negotiation successful
0926 13:47:14.202700 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8773,"thread_start_us":145,"threads_started":1}
I20260926 13:47:14.205090 25575 heartbeater.cc:347] Connected to a master server at 127.12.170.126:33809
I20260926 13:47:14.205338 25575 heartbeater.cc:464] Registering TS with master...
I20260926 13:47:14.205895 25575 heartbeater.cc:511] Master 127.12.170.126:33809 requested a full tablet report, sending...
I20260926 13:47:14.206956 25225 ts_manager.cc:194] Registered new tserver with Master: 9d067b92ac2d4b9f8ea3ffe2f51c8d9c (127.12.170.66:45035)
I20260926 13:47:14.207729 25225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.66@KRBTEST.COM'} at 127.12.170.66:40021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
I20260926 13:47:14.230835 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.170.84:38147/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:0
--local_ip_for_outbound_sockets=127.12.170.67
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:33809
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:46113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:47:14 dist-test-slave-hc1k krb5kdc[24749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430433, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for HTTP/127.12.170.84@KRBTEST.COM
W20260926 13:47:14.388398 25580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:14.388820 25580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:14.388922 25580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:14.394779 25580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:14.394915 25580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:14.395040 25580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:47:14.402545 25580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:46113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.170.84:38147/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:33809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:14.404024 25580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:14.405340 25580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:47:14.417852 25586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:14.418383 25585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:14.418673 25588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:14.419812 25580 server_base.cc:1034] running on GCE node
I20260926 13:47:14.420258 25580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:14.420969 25580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:14.424809 25580 hybrid_clock.cc:648] HybridClock initialized: now 1790430434424777 us; error 36 us; skew 500 ppm
Sep 26 13:47:14 dist-test-slave-hc1k krb5kdc[24749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430434, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:14.429227 25580 init.cc:377] Logged in from keytab as kudu/127.12.170.67@KRBTEST.COM (short username kudu)
I20260926 13:47:14.430318 25580 webserver.cc:460] Webserver started at http://127.12.170.67:42791/ using document root <none> and password file <none>
I20260926 13:47:14.430996 25580 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:14.431066 25580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:47:14.431265 25580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:47:14 dist-test-slave-hc1k krb5kdc[24749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430434, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for HTTP/127.12.170.84@KRBTEST.COM
I20260926 13:47:14.444520 25580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/data/instance:
uuid: "7f60a580e5cf4ba588c432ca69b7598b"
format_stamp: "Formatted at 2026-09-26 13:47:14 on dist-test-slave-hc1k"
server_key: "9d2d3d93bc6ca21b608b326a233eb79d"
server_key_iv: "712e0d0b2ef51999a15f3e5440e220d8"
server_key_version: "kuduclusterkey@0"
I20260926 13:47:14.445585 25580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance:
uuid: "7f60a580e5cf4ba588c432ca69b7598b"
format_stamp: "Formatted at 2026-09-26 13:47:14 on dist-test-slave-hc1k"
server_key: "9d2d3d93bc6ca21b608b326a233eb79d"
server_key_iv: "712e0d0b2ef51999a15f3e5440e220d8"
server_key_version: "kuduclusterkey@0"
I20260926 13:47:14.455368 25580 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260926 13:47:14.476712 25595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:14.478302 25580 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260926 13:47:14.478410 25580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "7f60a580e5cf4ba588c432ca69b7598b"
format_stamp: "Formatted at 2026-09-26 13:47:14 on dist-test-slave-hc1k"
server_key: "9d2d3d93bc6ca21b608b326a233eb79d"
server_key_iv: "712e0d0b2ef51999a15f3e5440e220d8"
server_key_version: "kuduclusterkey@0"
I20260926 13:47:14.478524 25580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:14.502575 25580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:14.506122 25580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:14.506342 25580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:14.506996 25580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:47:14.508031 25580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:47:14.508116 25580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:14.508193 25580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:47:14.508239 25580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:14.519454 25580 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:32861
I20260926 13:47:14.520598 25580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:47:14.524711 25708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:32861 every 8 connection(s)
I20260926 13:47:14.525036 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 25580
I20260926 13:47:14.525142 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance
I20260926 13:47:14.525425 12969 external_mini_cluster.cc:1428] Setting key b70717b9964688314aa1184009149db7
Sep 26 13:47:14 dist-test-slave-hc1k krb5kdc[24749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430434, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:47:14.546615 25439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:14.527623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:48853 (local address 127.12.170.126:33809)
0926 13:47:14.527769 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:14.527774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:14.532098 (+ 4324us) server_negotiation.cc:407] Connection header received
0926 13:47:14.533818 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:14.533822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:14.533870 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:14.533916 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:14.535488 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:14.536345 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:14.538117 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:14.538459 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:14.538527 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:14.542711 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:14.542714 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:14.542717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:14.542742 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:14.544393 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:14.544982 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:14.544986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:14.544987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:14.545068 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:14.545440 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:14.545443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:14.545445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:14.545648 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:14.545714 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:14.546348 (+ 634us) server_negotiation.cc:299] Negotiation successful
0926 13:47:14.546458 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 13:47:14.548336 25711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:14.527621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33809 (local address 127.12.170.67:48853)
0926 13:47:14.531948 (+ 4327us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:14.532012 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:14.533590 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:14.534337 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:14.534346 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:14.534681 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:14.535315 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:14.535332 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:14.536490 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:14.536493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:14.537974 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:14.537985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:14.540112 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:14.540115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:14.540307 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:14.540918 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:14.540938 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:14.542543 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:14.544525 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:14.544530 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:14.544544 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:14.544865 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:14.545166 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:14.545168 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:14.545170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:14.545335 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:14.545786 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:14.545792 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:14.546092 (+ 300us) client_negotiation.cc:769] Sending connection context
0926 13:47:14.547036 (+ 944us) client_negotiation.cc:240] Negotiation successful
0926 13:47:14.547345 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4072,"thread_start_us":127,"threads_started":1}
I20260926 13:47:14.549715 25709 heartbeater.cc:347] Connected to a master server at 127.12.170.126:33809
I20260926 13:47:14.549932 25709 heartbeater.cc:464] Registering TS with master...
I20260926 13:47:14.550453 25709 heartbeater.cc:511] Master 127.12.170.126:33809 requested a full tablet report, sending...
I20260926 13:47:14.552721 25225 ts_manager.cc:194] Registered new tserver with Master: 7f60a580e5cf4ba588c432ca69b7598b (127.12.170.67:32861)
I20260926 13:47:14.553409 25225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.67@KRBTEST.COM'} at 127.12.170.67:48853
I20260926 13:47:14.563575 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:47:14.564312 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 25306
I20260926 13:47:14.571297 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 25445
I20260926 13:47:14.578644 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 25580
I20260926 13:47:14.585144 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 25193
I20260926 13:47:14.590274 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:33809
--webserver_interface=127.12.170.126
--webserver_port=42985
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:46113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:33809
--ranger_config_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.170.84:38147/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.170.84:38147/kms with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:47:14.693499 25712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:14.693725 25712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:14.693761 25712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:14.697599 25712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:47:14.697645 25712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:14.697662 25712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:14.697676 25712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:47:14.697690 25712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:47:14.702733 25712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:46113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.170.84:38147/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:33809
--ranger_config_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:33809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=42985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:14.703569 25712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:14.704437 25712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:47:14.711970 25720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:14.712052 25718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:14.712121 25712 server_base.cc:1034] running on GCE node
W20260926 13:47:14.712029 25717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:14.712651 25712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:14.713624 25712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:14.714779 25712 hybrid_clock.cc:648] HybridClock initialized: now 1790430434714754 us; error 36 us; skew 500 ppm
Sep 26 13:47:14 dist-test-slave-hc1k krb5kdc[24749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430434, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:14.717331 25712 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:47:14.718138 25712 webserver.cc:460] Webserver started at http://127.12.170.126:42985/ using document root <none> and password file <none>
I20260926 13:47:14.718601 25712 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:14.718665 25712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 13:47:14 dist-test-slave-hc1k krb5kdc[24749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430434, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for HTTP/127.12.170.84@KRBTEST.COM
I20260926 13:47:14.785454 25712 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:47:14.788167 25728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:14.789644 25712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:47:14.789783 25712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "dceb4d13591a4cb2961bd52aa628cf7f"
format_stamp: "Formatted at 2026-09-26 13:47:12 on dist-test-slave-hc1k"
server_key: "98ce42c7d1a8fde0732aec15a0a34f3c"
server_key_iv: "b8596ac74bc4b59f2cf98e6623c34434"
server_key_version: "kuduclusterkey@0"
I20260926 13:47:14.790503 25712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:14.811798 25712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:14.814302 25712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:14.814438 25712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:14.820989 25712 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:33809
I20260926 13:47:14.821020 25780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:33809 every 8 connection(s)
I20260926 13:47:14.821966 25712 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:47:14.826033 25781 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:47:14.826645 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 25712
I20260926 13:47:14.827096 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.170.84:38147/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:44653
--local_ip_for_outbound_sockets=127.12.170.65
--tserver_master_addrs=127.12.170.126:33809
--webserver_port=34539
--webserver_interface=127.12.170.65
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:46113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.170.84:38147/kms with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:47:14.827936 25781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f: Bootstrap starting.
I20260926 13:47:14.843748 25781 log.cc:824] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f: Log is configured to *not* fsync() on all Append() calls
I20260926 13:47:14.849879 25781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:47:14.850220 25781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f: Bootstrap complete.
I20260926 13:47:14.854040 25781 raft_consensus.cc:348] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dceb4d13591a4cb2961bd52aa628cf7f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33809 } }
I20260926 13:47:14.854398 25781 raft_consensus.cc:729] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dceb4d13591a4cb2961bd52aa628cf7f, State: Initialized, Role: FOLLOWER
I20260926 13:47:14.854832 25781 consensus_queue.cc:260] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dceb4d13591a4cb2961bd52aa628cf7f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33809 } }
I20260926 13:47:14.854938 25781 raft_consensus.cc:388] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:47:14.855006 25781 raft_consensus.cc:482] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:47:14.855096 25781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [term 1 FOLLOWER]: Advancing to term 2
I20260926 13:47:14.856858 25781 raft_consensus.cc:504] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dceb4d13591a4cb2961bd52aa628cf7f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33809 } }
I20260926 13:47:14.857184 25781 leader_election.cc:302] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dceb4d13591a4cb2961bd52aa628cf7f; no voters:
I20260926 13:47:14.857496 25781 leader_election.cc:288] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 13:47:14.857962 25786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [term 2 FOLLOWER]: Leader election won for term 2
I20260926 13:47:14.858269 25786 raft_consensus.cc:686] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [term 2 LEADER]: Becoming Leader. State: Replica: dceb4d13591a4cb2961bd52aa628cf7f, State: Running, Role: LEADER
I20260926 13:47:14.858621 25781 sys_catalog.cc:564] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:47:14.858620 25786 consensus_queue.cc:237] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dceb4d13591a4cb2961bd52aa628cf7f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33809 } }
I20260926 13:47:14.860097 25787 sys_catalog.cc:455] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader dceb4d13591a4cb2961bd52aa628cf7f. Latest consensus state: current_term: 2 leader_uuid: "dceb4d13591a4cb2961bd52aa628cf7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dceb4d13591a4cb2961bd52aa628cf7f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33809 } } }
I20260926 13:47:14.860219 25787 sys_catalog.cc:458] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [sys.catalog]: This master's current role is: LEADER
I20260926 13:47:14.860469 25785 sys_catalog.cc:455] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dceb4d13591a4cb2961bd52aa628cf7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dceb4d13591a4cb2961bd52aa628cf7f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33809 } } }
I20260926 13:47:14.860567 25785 sys_catalog.cc:458] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f [sys.catalog]: This master's current role is: LEADER
I20260926 13:47:14.860777 25781 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 13:47:14.863276 25799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:47:14.867108 25799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:47:14.867792 25799 catalog_manager.cc:1259] Loaded cluster ID: a720acb978504118afdec5bef537fda8
I20260926 13:47:14.867875 25799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:47:14.870375 25799 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:47:14.871064 25799 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f: Loaded TSK: 0
I20260926 13:47:14.872318 25799 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 13:47:14.940765 25783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:14.941076 25783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:14.941186 25783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:14.947111 25783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:14.947176 25783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:14.947256 25783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:47:14.954763 25783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:46113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.170.84:38147/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:44653
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=34539
--tserver_master_addrs=127.12.170.126:33809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:14.956103 25783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:14.957322 25783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:47:14.968853 25821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:14.969141 25783 server_base.cc:1034] running on GCE node
W20260926 13:47:14.968863 25819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:14.969411 25818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:14.969841 25783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:14.970569 25783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:14.971732 25783 hybrid_clock.cc:648] HybridClock initialized: now 1790430434971712 us; error 35 us; skew 500 ppm
Sep 26 13:47:14 dist-test-slave-hc1k krb5kdc[24749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430434, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:14.975014 25783 init.cc:377] Logged in from keytab as kudu/127.12.170.65@KRBTEST.COM (short username kudu)
I20260926 13:47:14.976037 25783 webserver.cc:460] Webserver started at http://127.12.170.65:34539/ using document root <none> and password file <none>
I20260926 13:47:14.976715 25783 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:14.976814 25783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 13:47:14 dist-test-slave-hc1k krb5kdc[24749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430434, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for HTTP/127.12.170.84@KRBTEST.COM
I20260926 13:47:15.004483 25783 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260926 13:47:15.007730 25829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:15.009317 25783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:47:15.009442 25783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "182083588c30417e8a1b7c8360e17248"
format_stamp: "Formatted at 2026-09-26 13:47:13 on dist-test-slave-hc1k"
server_key: "0adf18b1a3a945e9da0a83e264d73514"
server_key_iv: "00bda0c08a77394a16d478f08e7a162f"
server_key_version: "kuduclusterkey@0"
I20260926 13:47:15.010169 25783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:15.029132 25783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:15.032481 25783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:15.032733 25783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:15.033406 25783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:47:15.034425 25783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:47:15.034561 25783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:15.034646 25783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:47:15.034693 25783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:15.047011 25783 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:44653
I20260926 13:47:15.047204 25942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:44653 every 8 connection(s)
I20260926 13:47:15.048220 25783 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:47:15.053634 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 25783
I20260926 13:47:15.054105 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.170.84:38147/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:45035
--local_ip_for_outbound_sockets=127.12.170.66
--tserver_master_addrs=127.12.170.126:33809
--webserver_port=45877
--webserver_interface=127.12.170.66
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:46113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.170.84:38147/kms with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:47:15 dist-test-slave-hc1k krb5kdc[24749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430434, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:47:15.075704 25945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:15.052211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33809 (local address 127.12.170.65:38267)
0926 13:47:15.052770 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:15.052826 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:15.053847 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:15.057885 (+ 4038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:15.057894 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:15.058234 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:15.059048 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:15.059070 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:15.061076 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:15.061080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:15.062407 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:15.062419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:15.063459 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:15.063463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:15.063696 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:15.068119 (+ 4423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:15.068140 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:15.069806 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:15.072242 (+ 2436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:15.072248 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:15.072263 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:15.072605 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:15.072984 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:15.072988 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:15.072990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:15.073143 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:15.073875 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:15.073881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:15.074182 (+ 301us) client_negotiation.cc:769] Sending connection context
0926 13:47:15.074373 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 13:47:15.074690 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":136,"threads_started":1}
I20260926 13:47:15.077492 25947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:15.055872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:38267 (local address 127.12.170.126:33809)
0926 13:47:15.056659 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:15.056681 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:15.056724 (+ 43us) server_negotiation.cc:407] Connection header received
0926 13:47:15.057014 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:15.057031 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:15.057447 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:15.057781 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:15.059239 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:15.060906 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:15.062558 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:15.063275 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:15.063439 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:15.070085 (+ 6646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:15.070092 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:15.070109 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:15.070144 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:15.072132 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:15.072744 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:15.072750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:15.072756 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:15.072890 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:15.073238 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:15.073241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:15.073243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:15.073691 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:15.073806 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:15.076361 (+ 2555us) server_negotiation.cc:299] Negotiation successful
0926 13:47:15.076643 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":120,"threads_started":1}
I20260926 13:47:15.080248 25943 heartbeater.cc:347] Connected to a master server at 127.12.170.126:33809
I20260926 13:47:15.080466 25943 heartbeater.cc:464] Registering TS with master...
I20260926 13:47:15.080976 25943 heartbeater.cc:511] Master 127.12.170.126:33809 requested a full tablet report, sending...
W20260926 13:47:15.083030 25943 heartbeater.cc:644] Failed to heartbeat to 127.12.170.126:33809 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 13:47:15.083149 25943 heartbeater.cc:464] Registering TS with master...
I20260926 13:47:15.083302 25943 heartbeater.cc:511] Master 127.12.170.126:33809 requested a full tablet report, sending...
I20260926 13:47:15.084062 25943 heartbeater.cc:464] Registering TS with master...
I20260926 13:47:15.084218 25943 heartbeater.cc:511] Master 127.12.170.126:33809 requested a full tablet report, sending...
W20260926 13:47:15.084972 25943 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:47:15.196810 25946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:15.197085 25946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:15.197171 25946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:15.201308 25946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:15.201416 25946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:15.201515 25946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:47:15.207309 25946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:46113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.170.84:38147/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:45035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=45877
--tserver_master_addrs=127.12.170.126:33809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:15.208353 25946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:15.209314 25946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:47:15.218360 25952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:15.218478 25953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:15.218721 25946 server_base.cc:1034] running on GCE node
W20260926 13:47:15.218935 25955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:15.219365 25946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:15.220005 25946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:15.221174 25946 hybrid_clock.cc:648] HybridClock initialized: now 1790430435221147 us; error 38 us; skew 500 ppm
Sep 26 13:47:15 dist-test-slave-hc1k krb5kdc[24749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430435, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:15.223770 25946 init.cc:377] Logged in from keytab as kudu/127.12.170.66@KRBTEST.COM (short username kudu)
I20260926 13:47:15.224608 25946 webserver.cc:460] Webserver started at http://127.12.170.66:45877/ using document root <none> and password file <none>
I20260926 13:47:15.225152 25946 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:15.225229 25946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 13:47:15 dist-test-slave-hc1k krb5kdc[24749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430435, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for HTTP/127.12.170.84@KRBTEST.COM
I20260926 13:47:15.249843 25946 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.000s
I20260926 13:47:15.253137 25963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:15.254804 25946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:47:15.254926 25946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "9d067b92ac2d4b9f8ea3ffe2f51c8d9c"
format_stamp: "Formatted at 2026-09-26 13:47:14 on dist-test-slave-hc1k"
server_key: "4af6c01b82cd9be34b0e18be5a8fb5a4"
server_key_iv: "ba7ce1cc2e64ce37382d12af201a533e"
server_key_version: "kuduclusterkey@0"
I20260926 13:47:15.255788 25946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:15.293871 25946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:15.297490 25946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:15.297746 25946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:15.298450 25946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:47:15.299470 25946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:47:15.299531 25946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:15.299585 25946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:47:15.299690 25946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:15.310652 25946 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:45035
I20260926 13:47:15.312131 25946 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:47:15.313169 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 25946
I20260926 13:47:15.313623 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.170.84:38147/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:32861
--local_ip_for_outbound_sockets=127.12.170.67
--tserver_master_addrs=127.12.170.126:33809
--webserver_port=42791
--webserver_interface=127.12.170.67
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:46113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.170.84:38147/kms with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:47:15.321650 26076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:45035 every 8 connection(s)
Sep 26 13:47:15 dist-test-slave-hc1k krb5kdc[24749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430435, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:47:15.330158 25947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:15.316271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:49801 (local address 127.12.170.126:33809)
0926 13:47:15.316417 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:15.316423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:15.317109 (+ 686us) server_negotiation.cc:407] Connection header received
0926 13:47:15.318427 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:15.318432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:15.318511 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:15.318577 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:15.320347 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:15.321158 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:15.323155 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:15.323522 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:15.323608 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:15.326486 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:15.326489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:15.326491 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:15.326516 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:15.327955 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:15.328436 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:15.328439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:15.328440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:15.328521 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:15.328896 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:15.328899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:15.328901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:15.329129 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:15.329212 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:15.329841 (+ 629us) server_negotiation.cc:299] Negotiation successful
0926 13:47:15.329994 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 13:47:15.331940 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:15.314901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33809 (local address 127.12.170.66:49801)
0926 13:47:15.316935 (+ 2034us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:15.316995 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:15.318159 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:15.319054 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:15.319065 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:15.319426 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:15.320149 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:15.320174 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:15.321345 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:15.321349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:15.323003 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:15.323016 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:15.323854 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:15.323859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:15.324064 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:15.324814 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:15.324837 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:15.326302 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:15.328064 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:15.328070 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:15.328084 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:15.328333 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:15.328598 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:15.328601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:15.328603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:15.328779 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:15.329278 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:15.329284 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:15.329563 (+ 279us) client_negotiation.cc:769] Sending connection context
0926 13:47:15.330576 (+ 1013us) client_negotiation.cc:240] Negotiation successful
0926 13:47:15.330892 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1798,"thread_start_us":155,"threads_started":1}
I20260926 13:47:15.333467 26077 heartbeater.cc:347] Connected to a master server at 127.12.170.126:33809
I20260926 13:47:15.333719 26077 heartbeater.cc:464] Registering TS with master...
I20260926 13:47:15.334300 26077 heartbeater.cc:511] Master 127.12.170.126:33809 requested a full tablet report, sending...
W20260926 13:47:15.335979 26077 heartbeater.cc:644] Failed to heartbeat to 127.12.170.126:33809 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 13:47:15.336122 26077 heartbeater.cc:464] Registering TS with master...
I20260926 13:47:15.336293 26077 heartbeater.cc:511] Master 127.12.170.126:33809 requested a full tablet report, sending...
I20260926 13:47:15.337110 26077 heartbeater.cc:464] Registering TS with master...
I20260926 13:47:15.337281 26077 heartbeater.cc:511] Master 127.12.170.126:33809 requested a full tablet report, sending...
W20260926 13:47:15.338001 26077 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:47:15.475728 26079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:15.476059 26079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:15.476121 26079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:15.481911 26079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:15.481989 26079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:15.482074 26079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:47:15.489640 26079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:46113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.170.84:38147/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:32861
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=42791
--tserver_master_addrs=127.12.170.126:33809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:15.490937 26079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:15.492275 26079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:47:15.504796 26086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:15.505002 26089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:15.505820 26079 server_base.cc:1034] running on GCE node
W20260926 13:47:15.507774 26087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:15.508278 26079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:15.509047 26079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:15.510236 26079 hybrid_clock.cc:648] HybridClock initialized: now 1790430435510208 us; error 40 us; skew 500 ppm
Sep 26 13:47:15 dist-test-slave-hc1k krb5kdc[24749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430435, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:15.513814 26079 init.cc:377] Logged in from keytab as kudu/127.12.170.67@KRBTEST.COM (short username kudu)
I20260926 13:47:15.514824 26079 webserver.cc:460] Webserver started at http://127.12.170.67:42791/ using document root <none> and password file <none>
I20260926 13:47:15.515532 26079 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:15.515640 26079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 13:47:15 dist-test-slave-hc1k krb5kdc[24749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430435, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for HTTP/127.12.170.84@KRBTEST.COM
I20260926 13:47:15.542199 26079 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 13:47:15.545305 26097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:15.546916 26079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:47:15.547049 26079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "7f60a580e5cf4ba588c432ca69b7598b"
format_stamp: "Formatted at 2026-09-26 13:47:14 on dist-test-slave-hc1k"
server_key: "9d2d3d93bc6ca21b608b326a233eb79d"
server_key_iv: "712e0d0b2ef51999a15f3e5440e220d8"
server_key_version: "kuduclusterkey@0"
I20260926 13:47:15.547780 26079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:15.566709 26079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:15.570370 26079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:15.570544 26079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:15.571201 26079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:47:15.572243 26079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:47:15.572292 26079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:15.572342 26079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:47:15.572363 26079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:15.587213 26079 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:32861
I20260926 13:47:15.588526 26079 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:47:15.590747 26210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:32861 every 8 connection(s)
I20260926 13:47:15.596468 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 26079
Sep 26 13:47:15 dist-test-slave-hc1k krb5kdc[24749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430435, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:47:15.604570 25947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:15.592524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:52885 (local address 127.12.170.126:33809)
0926 13:47:15.592672 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:15.592677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:15.593531 (+ 854us) server_negotiation.cc:407] Connection header received
0926 13:47:15.594486 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:15.594490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:15.594547 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:15.594670 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:15.595710 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:15.596506 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:15.598005 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:15.598241 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:15.598325 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:15.600643 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:15.600646 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:15.600648 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:15.600669 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:15.602244 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:15.602879 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:15.602883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:15.602885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:15.602959 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:15.603331 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:15.603335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:15.603337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:15.603564 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:15.603698 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:15.604302 (+ 604us) server_negotiation.cc:299] Negotiation successful
0926 13:47:15.604407 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 13:47:15.605901 26214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:15.597116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40824 (local address 127.12.170.126:33809)
0926 13:47:15.597692 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:15.597696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:15.597707 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:47:15.597742 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:15.597745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:15.597785 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:15.597866 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:15.598791 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:15.599512 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:15.600792 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:15.601076 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:15.601168 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:15.602690 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:15.602693 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:15.602695 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:15.602719 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:15.604250 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:15.604672 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:15.604675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:15.604677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:15.604770 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:15.605039 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:15.605042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:15.605043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:15.605241 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:15.605435 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:15.605650 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 13:47:15.605758 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":75,"threads_started":1}
I20260926 13:47:15.606397 26213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:15.592913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33809 (local address 127.12.170.67:52885)
0926 13:47:15.593364 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:15.593415 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:15.594228 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:15.594806 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:15.594812 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:15.595072 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:15.595555 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:15.595568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:15.596659 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:15.596663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:15.597910 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:15.597919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:15.598384 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:15.598388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:15.598553 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:15.599078 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:15.599097 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:15.600495 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:15.602366 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:15.602372 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:15.602388 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:15.602777 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:15.603046 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:15.603050 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:15.603052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:15.603222 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:15.603727 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:15.603734 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:15.604032 (+ 298us) client_negotiation.cc:769] Sending connection context
0926 13:47:15.605037 (+ 1005us) client_negotiation.cc:240] Negotiation successful
0926 13:47:15.605410 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":111,"threads_started":1}
I20260926 13:47:15.607497 26211 heartbeater.cc:347] Connected to a master server at 127.12.170.126:33809
I20260926 13:47:15.607741 26211 heartbeater.cc:464] Registering TS with master...
I20260926 13:47:15.608282 26211 heartbeater.cc:511] Master 127.12.170.126:33809 requested a full tablet report, sending...
W20260926 13:47:15.609769 26211 heartbeater.cc:644] Failed to heartbeat to 127.12.170.126:33809 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 13:47:15.609894 26211 heartbeater.cc:464] Registering TS with master...
I20260926 13:47:15.610054 26211 heartbeater.cc:511] Master 127.12.170.126:33809 requested a full tablet report, sending...
I20260926 13:47:15.610877 26211 heartbeater.cc:464] Registering TS with master...
I20260926 13:47:15.611037 26211 heartbeater.cc:511] Master 127.12.170.126:33809 requested a full tablet report, sending...
W20260926 13:47:15.611796 26211 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 13:47:15 dist-test-slave-hc1k krb5kdc[24749](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790430435, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:16.085198 25943 heartbeater.cc:464] Registering TS with master...
I20260926 13:47:16.085520 25943 heartbeater.cc:511] Master 127.12.170.126:33809 requested a full tablet report, sending...
I20260926 13:47:16.086764 25744 ts_manager.cc:194] Registered new tserver with Master: 182083588c30417e8a1b7c8360e17248 (127.12.170.65:44653)
I20260926 13:47:16.088486 25744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.65@KRBTEST.COM'} at 127.12.170.65:38267
Sep 26 13:47:16 dist-test-slave-hc1k krb5kdc[24749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430435, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for HTTP/127.12.170.84@KRBTEST.COM
I20260926 13:47:16.338227 26077 heartbeater.cc:464] Registering TS with master...
I20260926 13:47:16.338523 26077 heartbeater.cc:511] Master 127.12.170.126:33809 requested a full tablet report, sending...
I20260926 13:47:16.339218 25744 ts_manager.cc:194] Registered new tserver with Master: 9d067b92ac2d4b9f8ea3ffe2f51c8d9c (127.12.170.66:45035)
I20260926 13:47:16.340018 25744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.66@KRBTEST.COM'} at 127.12.170.66:49801
Sep 26 13:47:16 dist-test-slave-hc1k krb5kdc[24749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430435, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for HTTP/127.12.170.84@KRBTEST.COM
I20260926 13:47:16.612012 26211 heartbeater.cc:464] Registering TS with master...
I20260926 13:47:16.612313 26211 heartbeater.cc:511] Master 127.12.170.126:33809 requested a full tablet report, sending...
I20260926 13:47:16.613170 25744 ts_manager.cc:194] Registered new tserver with Master: 7f60a580e5cf4ba588c432ca69b7598b (127.12.170.67:32861)
I20260926 13:47:16.613940 25744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.67@KRBTEST.COM'} at 127.12.170.67:52885
I20260926 13:47:16.617179 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:47:16.649489 26234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:16.623996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40840 (local address 127.12.170.126:33809)
0926 13:47:16.626089 (+ 2093us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:16.626094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:16.640023 (+ 13929us) server_negotiation.cc:407] Connection header received
0926 13:47:16.640162 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:16.640165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:16.640230 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:16.640341 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:16.641349 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:16.642164 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:16.643313 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:16.643710 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:16.643815 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:16.645577 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:16.645581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:16.645585 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:16.645613 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:16.647298 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:16.647810 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:16.647814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:16.647815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:16.647905 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:16.648194 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:16.648197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:16.648199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:16.648414 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:16.648553 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:16.649173 (+ 620us) server_negotiation.cc:299] Negotiation successful
0926 13:47:16.649294 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2009,"thread_start_us":104,"threads_started":1}
I20260926 13:47:16.653748 25744 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:47:16.658293 25744 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:47:16.700753 26244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:16.684945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:44653 (local address 127.0.0.1:55570)
0926 13:47:16.694416 (+ 9471us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:16.694430 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:16.694522 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:16.695430 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:16.695433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:16.695452 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:47:16.695559 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:16.695566 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:16.697704 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:16.697708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:16.698861 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:16.698872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:16.700187 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:16.700191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:16.700305 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:16.700347 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 13:47:16.700471 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 13:47:16.700593 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9411,"thread_start_us":85,"threads_started":1}
I20260926 13:47:16.701339 26240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:16.682675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:32861 (local address 127.0.0.1:55792)
0926 13:47:16.687917 (+ 5242us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:16.687971 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:16.688094 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:16.689009 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:16.689015 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:16.689040 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:47:16.689164 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:16.689170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:16.694258 (+ 5088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:16.694261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:16.695646 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:16.695688 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:16.700998 (+ 5310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:16.701002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:16.701090 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:16.701112 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 13:47:16.701155 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 13:47:16.701215 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5060,"thread_start_us":113,"threads_started":1}
I20260926 13:47:16.701859 26245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:16.685122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55570 (local address 127.12.170.65:44653)
0926 13:47:16.694772 (+ 9650us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:16.694779 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:16.694803 (+ 24us) server_negotiation.cc:407] Connection header received
0926 13:47:16.694873 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:16.694878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:16.695052 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:16.695295 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:47:16.695708 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:16.696861 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:16.699004 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:16.699877 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:16.699978 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:16.700124 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:16.701389 (+ 1265us) server_negotiation.cc:299] Negotiation successful
0926 13:47:16.701717 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9541,"thread_start_us":105,"threads_started":1}
I20260926 13:47:16.702229 26241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:16.682775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55792 (local address 127.12.170.67:32861)
0926 13:47:16.688363 (+ 5588us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:16.688371 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:16.688395 (+ 24us) server_negotiation.cc:407] Connection header received
0926 13:47:16.688468 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:16.688473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:16.688691 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:16.688831 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:47:16.691299 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:16.694156 (+ 2857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:16.695806 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:16.696615 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:16.701807 (+ 5192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:16.701952 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:16.702002 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 13:47:16.702095 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5481,"thread_start_us":90,"threads_started":1}
I20260926 13:47:16.705643 26242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:16.683835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:45035 (local address 127.0.0.1:48032)
0926 13:47:16.689354 (+ 5519us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:16.689365 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:16.689434 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:16.693812 (+ 4378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:16.693815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:16.693835 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:47:16.693930 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:16.693935 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:16.703154 (+ 9219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:16.703158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:16.704239 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:16.704249 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:16.705299 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:16.705303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:16.705382 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:16.705404 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 13:47:16.705447 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 13:47:16.705506 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5463,"thread_start_us":92,"threads_started":1}
I20260926 13:47:16.705220 26138 tablet_service.cc:1467] Processing CreateTablet for tablet 3b9fefa6f6cc4791801b422fb898bb0d (DEFAULT_TABLE table=test-table [id=56fee80d7e434616b44faab0b0123034]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:47:16.705775 25877 tablet_service.cc:1467] Processing CreateTablet for tablet 3b9fefa6f6cc4791801b422fb898bb0d (DEFAULT_TABLE table=test-table [id=56fee80d7e434616b44faab0b0123034]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:47:16.706367 26138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b9fefa6f6cc4791801b422fb898bb0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:47:16.706426 26243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:16.684237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48032 (local address 127.12.170.66:45035)
0926 13:47:16.693345 (+ 9108us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:16.693352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:16.693377 (+ 25us) server_negotiation.cc:407] Connection header received
0926 13:47:16.693450 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:16.693467 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:16.693681 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:16.693841 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:47:16.699742 (+ 5901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:16.700854 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:16.704390 (+ 3536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:16.705167 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:16.706037 (+ 870us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:16.706170 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:16.706217 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 13:47:16.706298 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8995,"thread_start_us":111,"threads_started":1}
I20260926 13:47:16.706741 25877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b9fefa6f6cc4791801b422fb898bb0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:47:16.708870 25995 tablet_service.cc:1467] Processing CreateTablet for tablet 3b9fefa6f6cc4791801b422fb898bb0d (DEFAULT_TABLE table=test-table [id=56fee80d7e434616b44faab0b0123034]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:47:16.709779 25995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b9fefa6f6cc4791801b422fb898bb0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:47:16.716758 26247 tablet_bootstrap.cc:492] T 3b9fefa6f6cc4791801b422fb898bb0d P 182083588c30417e8a1b7c8360e17248: Bootstrap starting.
I20260926 13:47:16.719218 26247 tablet_bootstrap.cc:654] T 3b9fefa6f6cc4791801b422fb898bb0d P 182083588c30417e8a1b7c8360e17248: Neither blocks nor log segments found. Creating new log.
I20260926 13:47:16.721516 26247 log.cc:824] T 3b9fefa6f6cc4791801b422fb898bb0d P 182083588c30417e8a1b7c8360e17248: Log is configured to *not* fsync() on all Append() calls
I20260926 13:47:16.722257 26248 tablet_bootstrap.cc:492] T 3b9fefa6f6cc4791801b422fb898bb0d P 9d067b92ac2d4b9f8ea3ffe2f51c8d9c: Bootstrap starting.
I20260926 13:47:16.722680 26246 tablet_bootstrap.cc:492] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b: Bootstrap starting.
I20260926 13:47:16.724934 26248 tablet_bootstrap.cc:654] T 3b9fefa6f6cc4791801b422fb898bb0d P 9d067b92ac2d4b9f8ea3ffe2f51c8d9c: Neither blocks nor log segments found. Creating new log.
I20260926 13:47:16.726954 26246 tablet_bootstrap.cc:654] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b: Neither blocks nor log segments found. Creating new log.
I20260926 13:47:16.727859 26248 log.cc:824] T 3b9fefa6f6cc4791801b422fb898bb0d P 9d067b92ac2d4b9f8ea3ffe2f51c8d9c: Log is configured to *not* fsync() on all Append() calls
I20260926 13:47:16.728664 26246 log.cc:824] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b: Log is configured to *not* fsync() on all Append() calls
I20260926 13:47:16.731619 26246 tablet_bootstrap.cc:492] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b: No bootstrap required, opened a new log
I20260926 13:47:16.731838 26246 ts_tablet_manager.cc:1397] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260926 13:47:16.735728 26246 raft_consensus.cc:348] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f60a580e5cf4ba588c432ca69b7598b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 32861 } } peers { permanent_uuid: "9d067b92ac2d4b9f8ea3ffe2f51c8d9c" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45035 } } peers { permanent_uuid: "182083588c30417e8a1b7c8360e17248" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 44653 } }
I20260926 13:47:16.736003 26246 raft_consensus.cc:374] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:47:16.736065 26246 raft_consensus.cc:729] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f60a580e5cf4ba588c432ca69b7598b, State: Initialized, Role: FOLLOWER
I20260926 13:47:16.738314 26247 tablet_bootstrap.cc:492] T 3b9fefa6f6cc4791801b422fb898bb0d P 182083588c30417e8a1b7c8360e17248: No bootstrap required, opened a new log
I20260926 13:47:16.738521 26247 ts_tablet_manager.cc:1397] T 3b9fefa6f6cc4791801b422fb898bb0d P 182083588c30417e8a1b7c8360e17248: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20260926 13:47:16.739130 26248 tablet_bootstrap.cc:492] T 3b9fefa6f6cc4791801b422fb898bb0d P 9d067b92ac2d4b9f8ea3ffe2f51c8d9c: No bootstrap required, opened a new log
I20260926 13:47:16.739343 26248 ts_tablet_manager.cc:1397] T 3b9fefa6f6cc4791801b422fb898bb0d P 9d067b92ac2d4b9f8ea3ffe2f51c8d9c: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20260926 13:47:16.742316 26247 raft_consensus.cc:348] T 3b9fefa6f6cc4791801b422fb898bb0d P 182083588c30417e8a1b7c8360e17248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f60a580e5cf4ba588c432ca69b7598b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 32861 } } peers { permanent_uuid: "9d067b92ac2d4b9f8ea3ffe2f51c8d9c" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45035 } } peers { permanent_uuid: "182083588c30417e8a1b7c8360e17248" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 44653 } }
I20260926 13:47:16.742581 26247 raft_consensus.cc:374] T 3b9fefa6f6cc4791801b422fb898bb0d P 182083588c30417e8a1b7c8360e17248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:47:16.743145 26247 raft_consensus.cc:729] T 3b9fefa6f6cc4791801b422fb898bb0d P 182083588c30417e8a1b7c8360e17248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182083588c30417e8a1b7c8360e17248, State: Initialized, Role: FOLLOWER
I20260926 13:47:16.743638 26247 consensus_queue.cc:260] T 3b9fefa6f6cc4791801b422fb898bb0d P 182083588c30417e8a1b7c8360e17248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f60a580e5cf4ba588c432ca69b7598b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 32861 } } peers { permanent_uuid: "9d067b92ac2d4b9f8ea3ffe2f51c8d9c" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45035 } } peers { permanent_uuid: "182083588c30417e8a1b7c8360e17248" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 44653 } }
I20260926 13:47:16.743707 26248 raft_consensus.cc:348] T 3b9fefa6f6cc4791801b422fb898bb0d P 9d067b92ac2d4b9f8ea3ffe2f51c8d9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f60a580e5cf4ba588c432ca69b7598b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 32861 } } peers { permanent_uuid: "9d067b92ac2d4b9f8ea3ffe2f51c8d9c" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45035 } } peers { permanent_uuid: "182083588c30417e8a1b7c8360e17248" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 44653 } }
I20260926 13:47:16.744072 26248 raft_consensus.cc:374] T 3b9fefa6f6cc4791801b422fb898bb0d P 9d067b92ac2d4b9f8ea3ffe2f51c8d9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:47:16.744161 26248 raft_consensus.cc:729] T 3b9fefa6f6cc4791801b422fb898bb0d P 9d067b92ac2d4b9f8ea3ffe2f51c8d9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d067b92ac2d4b9f8ea3ffe2f51c8d9c, State: Initialized, Role: FOLLOWER
I20260926 13:47:16.744668 26247 ts_tablet_manager.cc:1428] T 3b9fefa6f6cc4791801b422fb898bb0d P 182083588c30417e8a1b7c8360e17248: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260926 13:47:16.744650 26248 consensus_queue.cc:260] T 3b9fefa6f6cc4791801b422fb898bb0d P 9d067b92ac2d4b9f8ea3ffe2f51c8d9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f60a580e5cf4ba588c432ca69b7598b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 32861 } } peers { permanent_uuid: "9d067b92ac2d4b9f8ea3ffe2f51c8d9c" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45035 } } peers { permanent_uuid: "182083588c30417e8a1b7c8360e17248" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 44653 } }
I20260926 13:47:16.745031 25943 heartbeater.cc:503] Master 127.12.170.126:33809 was elected leader, sending a full tablet report...
I20260926 13:47:16.745816 26248 ts_tablet_manager.cc:1428] T 3b9fefa6f6cc4791801b422fb898bb0d P 9d067b92ac2d4b9f8ea3ffe2f51c8d9c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 13:47:16.746137 26246 consensus_queue.cc:260] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f60a580e5cf4ba588c432ca69b7598b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 32861 } } peers { permanent_uuid: "9d067b92ac2d4b9f8ea3ffe2f51c8d9c" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45035 } } peers { permanent_uuid: "182083588c30417e8a1b7c8360e17248" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 44653 } }
I20260926 13:47:16.747066 26246 ts_tablet_manager.cc:1428] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260926 13:47:16.747431 26077 heartbeater.cc:503] Master 127.12.170.126:33809 was elected leader, sending a full tablet report...
I20260926 13:47:16.747907 26211 heartbeater.cc:503] Master 127.12.170.126:33809 was elected leader, sending a full tablet report...
W20260926 13:47:16.802052 25944 tablet.cc:2369] T 3b9fefa6f6cc4791801b422fb898bb0d P 182083588c30417e8a1b7c8360e17248: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:47:16.825980 26254 raft_consensus.cc:482] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:47:16.826198 26254 raft_consensus.cc:504] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f60a580e5cf4ba588c432ca69b7598b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 32861 } } peers { permanent_uuid: "9d067b92ac2d4b9f8ea3ffe2f51c8d9c" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45035 } } peers { permanent_uuid: "182083588c30417e8a1b7c8360e17248" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 44653 } }
W20260926 13:47:16.826905 26078 tablet.cc:2369] T 3b9fefa6f6cc4791801b422fb898bb0d P 9d067b92ac2d4b9f8ea3ffe2f51c8d9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:47:16.827504 26254 leader_election.cc:288] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d067b92ac2d4b9f8ea3ffe2f51c8d9c (127.12.170.66:45035), 182083588c30417e8a1b7c8360e17248 (127.12.170.65:44653)
I20260926 13:47:16.832055 26263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:16.827838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:44653 (local address 127.12.170.67:37199)
0926 13:47:16.828298 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:16.828309 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:16.828421 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:16.828775 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:16.828779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:16.828798 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:47:16.828978 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:16.828985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:16.829920 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:16.829924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:16.831027 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:16.831036 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:16.831622 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:16.831626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:16.831750 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:16.831769 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 13:47:16.831832 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 13:47:16.831894 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":201,"thread_start_us":130,"threads_started":1}
I20260926 13:47:16.832036 26245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:16.827933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:37199 (local address 127.12.170.65:44653)
0926 13:47:16.828050 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:16.828055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:16.828369 (+ 314us) server_negotiation.cc:407] Connection header received
0926 13:47:16.828556 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:16.828560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:16.828601 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:16.828666 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:47:16.829126 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:16.829827 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:16.831144 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:16.831522 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:16.831622 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:16.831697 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:16.831857 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 13:47:16.831926 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 13:47:16.832593 25887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9fefa6f6cc4791801b422fb898bb0d" candidate_uuid: "7f60a580e5cf4ba588c432ca69b7598b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182083588c30417e8a1b7c8360e17248" is_pre_election: true
I20260926 13:47:16.832718 26262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:16.827838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:45035 (local address 127.12.170.67:59861)
0926 13:47:16.828217 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:16.828243 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:16.828369 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:16.828665 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:16.828668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:16.828683 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:47:16.828832 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:16.828838 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:16.829804 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:16.829808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:16.831532 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:16.831542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:16.832333 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:16.832336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:16.832428 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:16.832449 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 13:47:16.832516 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 13:47:16.832586 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":2,"thread_start_us":123,"threads_started":1}
I20260926 13:47:16.832710 26243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:16.827851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:59861 (local address 127.12.170.66:45035)
0926 13:47:16.827985 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:16.827990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:16.828333 (+ 343us) server_negotiation.cc:407] Connection header received
0926 13:47:16.828507 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:16.828511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:16.828565 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:16.828646 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:47:16.828935 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:16.829704 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:16.831713 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:16.832139 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:16.832244 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:16.832292 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:16.832547 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 13:47:16.832615 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 13:47:16.832912 25887 raft_consensus.cc:2454] T 3b9fefa6f6cc4791801b422fb898bb0d P 182083588c30417e8a1b7c8360e17248 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f60a580e5cf4ba588c432ca69b7598b in term 0.
I20260926 13:47:16.833367 26098 leader_election.cc:302] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 182083588c30417e8a1b7c8360e17248, 7f60a580e5cf4ba588c432ca69b7598b; no voters:
I20260926 13:47:16.833366 26025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9fefa6f6cc4791801b422fb898bb0d" candidate_uuid: "7f60a580e5cf4ba588c432ca69b7598b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d067b92ac2d4b9f8ea3ffe2f51c8d9c" is_pre_election: true
I20260926 13:47:16.833609 26025 raft_consensus.cc:2454] T 3b9fefa6f6cc4791801b422fb898bb0d P 9d067b92ac2d4b9f8ea3ffe2f51c8d9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f60a580e5cf4ba588c432ca69b7598b in term 0.
I20260926 13:47:16.833642 26254 raft_consensus.cc:2781] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:47:16.833722 26254 raft_consensus.cc:482] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:47:16.833777 26254 raft_consensus.cc:3037] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:47:16.834846 26254 raft_consensus.cc:504] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f60a580e5cf4ba588c432ca69b7598b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 32861 } } peers { permanent_uuid: "9d067b92ac2d4b9f8ea3ffe2f51c8d9c" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45035 } } peers { permanent_uuid: "182083588c30417e8a1b7c8360e17248" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 44653 } }
I20260926 13:47:16.835170 26254 leader_election.cc:288] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b [CANDIDATE]: Term 1 election: Requested vote from peers 9d067b92ac2d4b9f8ea3ffe2f51c8d9c (127.12.170.66:45035), 182083588c30417e8a1b7c8360e17248 (127.12.170.65:44653)
I20260926 13:47:16.835429 26025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9fefa6f6cc4791801b422fb898bb0d" candidate_uuid: "7f60a580e5cf4ba588c432ca69b7598b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d067b92ac2d4b9f8ea3ffe2f51c8d9c"
I20260926 13:47:16.835546 25887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9fefa6f6cc4791801b422fb898bb0d" candidate_uuid: "7f60a580e5cf4ba588c432ca69b7598b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "182083588c30417e8a1b7c8360e17248"
I20260926 13:47:16.835563 26025 raft_consensus.cc:3037] T 3b9fefa6f6cc4791801b422fb898bb0d P 9d067b92ac2d4b9f8ea3ffe2f51c8d9c [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:47:16.835700 25887 raft_consensus.cc:3037] T 3b9fefa6f6cc4791801b422fb898bb0d P 182083588c30417e8a1b7c8360e17248 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:47:16.836621 26025 raft_consensus.cc:2454] T 3b9fefa6f6cc4791801b422fb898bb0d P 9d067b92ac2d4b9f8ea3ffe2f51c8d9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f60a580e5cf4ba588c432ca69b7598b in term 1.
I20260926 13:47:16.836666 25887 raft_consensus.cc:2454] T 3b9fefa6f6cc4791801b422fb898bb0d P 182083588c30417e8a1b7c8360e17248 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f60a580e5cf4ba588c432ca69b7598b in term 1.
I20260926 13:47:16.836882 26100 leader_election.cc:302] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f60a580e5cf4ba588c432ca69b7598b, 9d067b92ac2d4b9f8ea3ffe2f51c8d9c; no voters:
I20260926 13:47:16.837036 26254 raft_consensus.cc:2781] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:47:16.837218 26254 raft_consensus.cc:686] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b [term 1 LEADER]: Becoming Leader. State: Replica: 7f60a580e5cf4ba588c432ca69b7598b, State: Running, Role: LEADER
I20260926 13:47:16.837531 26254 consensus_queue.cc:237] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f60a580e5cf4ba588c432ca69b7598b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 32861 } } peers { permanent_uuid: "9d067b92ac2d4b9f8ea3ffe2f51c8d9c" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45035 } } peers { permanent_uuid: "182083588c30417e8a1b7c8360e17248" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 44653 } }
I20260926 13:47:16.840546 25743 catalog_manager.cc:5495] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f60a580e5cf4ba588c432ca69b7598b (127.12.170.67). New cstate: current_term: 1 leader_uuid: "7f60a580e5cf4ba588c432ca69b7598b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f60a580e5cf4ba588c432ca69b7598b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 32861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d067b92ac2d4b9f8ea3ffe2f51c8d9c" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "182083588c30417e8a1b7c8360e17248" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 44653 } health_report { overall_health: UNKNOWN } } }
W20260926 13:47:16.845593 26212 tablet.cc:2369] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:47:16.875188 26241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:16.872070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43376 (local address 127.12.170.67:32861)
0926 13:47:16.872182 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:16.872186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:16.872198 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:47:16.872286 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:16.872288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:16.872340 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:16.872423 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:47:16.872612 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:16.873127 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:16.873896 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:16.874115 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:16.874183 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:16.874445 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:47:16.874851 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:47:16.874932 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:16.874992 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 13:47:16.875041 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:47:16.880524 26025 raft_consensus.cc:1261] T 3b9fefa6f6cc4791801b422fb898bb0d P 9d067b92ac2d4b9f8ea3ffe2f51c8d9c [term 1 FOLLOWER]: Refusing update from remote peer 7f60a580e5cf4ba588c432ca69b7598b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:47:16.880525 25887 raft_consensus.cc:1261] T 3b9fefa6f6cc4791801b422fb898bb0d P 182083588c30417e8a1b7c8360e17248 [term 1 FOLLOWER]: Refusing update from remote peer 7f60a580e5cf4ba588c432ca69b7598b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:47:16.881307 26264 consensus_queue.cc:1035] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b [LEADER]: Connected to new peer: Peer: permanent_uuid: "182083588c30417e8a1b7c8360e17248" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 44653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:47:16.881469 26254 consensus_queue.cc:1035] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d067b92ac2d4b9f8ea3ffe2f51c8d9c" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:47:16.888161 26268 mvcc.cc:205] Tried to move back new op lower bound from 7333603069456052224 to 7333603069289185280. Current Snapshot: MvccSnapshot[applied={T|T < 7333603069456052224}]
I20260926 13:47:16.888401 26267 mvcc.cc:205] Tried to move back new op lower bound from 7333603069456052224 to 7333603069289185280. Current Snapshot: MvccSnapshot[applied={T|T < 7333603069456052224}]
I20260926 13:47:16.896173 25743 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:47:16.896309 25743 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:47:16.898422 25743 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dceb4d13591a4cb2961bd52aa628cf7f: Sending DeleteTablet for 3 replicas of tablet 3b9fefa6f6cc4791801b422fb898bb0d
I20260926 13:47:16.899191 25877 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b9fefa6f6cc4791801b422fb898bb0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:47:16 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:55570
I20260926 13:47:16.899299 26138 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b9fefa6f6cc4791801b422fb898bb0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:47:16 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:55792
I20260926 13:47:16.899684 25995 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b9fefa6f6cc4791801b422fb898bb0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:47:16 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:48032
I20260926 13:47:16.899822 26277 tablet_replica.cc:330] T 3b9fefa6f6cc4791801b422fb898bb0d P 182083588c30417e8a1b7c8360e17248: stopping tablet replica
I20260926 13:47:16.900115 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 25783
I20260926 13:47:16.900113 26277 raft_consensus.cc:2229] T 3b9fefa6f6cc4791801b422fb898bb0d P 182083588c30417e8a1b7c8360e17248 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:47:16.900379 26277 raft_consensus.cc:2258] T 3b9fefa6f6cc4791801b422fb898bb0d P 182083588c30417e8a1b7c8360e17248 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 13:47:16.906081 25729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.170.65:44653 (error 108)
I20260926 13:47:16.906275 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 25946
I20260926 13:47:16.906275 26278 tablet_replica.cc:330] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b: stopping tablet replica
I20260926 13:47:16.906538 26278 raft_consensus.cc:2229] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:47:16.906824 26278 raft_consensus.cc:2258] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:47:16.908394 26242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:16.908156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:44653 (local address 127.0.0.1:32988)
0926 13:47:16.908264 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:16.908324 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.65:44653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 13:47:16.908546 25729 catalog_manager.cc:4584] TS 182083588c30417e8a1b7c8360e17248 (127.12.170.65:44653): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3b9fefa6f6cc4791801b422fb898bb0d: Network error: Client connection negotiation failed: client connection to 127.12.170.65:44653: connect: Connection refused (error 111)
I20260926 13:47:16.908872 26278 ts_tablet_manager.cc:1905] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:47:16.911586 26278 ts_tablet_manager.cc:1918] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:47:16.911700 26278 log.cc:1196] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/3b9fefa6f6cc4791801b422fb898bb0d
I20260926 13:47:16.911981 26278 ts_tablet_manager.cc:1939] T 3b9fefa6f6cc4791801b422fb898bb0d P 7f60a580e5cf4ba588c432ca69b7598b: Deleting consensus metadata
I20260926 13:47:16.912469 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 26079
I20260926 13:47:16.912757 26242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:16.912497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:45035 (local address 127.0.0.1:39958)
0926 13:47:16.912647 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:16.912689 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.66:45035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 13:47:16.917018 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 25712
I20260926 13:47:16.924156 12969 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 13:47:17.258404 12969 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 13:47:17.258508 12969 mini_ranger.cc:66] Stopping Ranger...
I20260926 13:47:17.592726 12969 mini_ranger.cc:68] Stopped Ranger
2026-09-26 13:47:17.592 UTC [24779] LOG: received smart shutdown request
2026-09-26 13:47:17.596 UTC [24779] LOG: background worker "logical replication launcher" (PID 24788) exited with exit code 1
2026-09-26 13:47:17.596 UTC [24783] LOG: shutting down
2026-09-26 13:47:17.746 UTC [24779] LOG: database system is shut down
2026-09-26T13:47:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46677 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:47:17 dist-test-slave-hc1k krb5kdc[26289](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:47:17 dist-test-slave-hc1k krb5kdc[26289](info): set up 2 sockets
Sep 26 13:47:17 dist-test-slave-hc1k krb5kdc[26289](info): commencing operation
krb5kdc: starting...
W20260926 13:47:18.799630 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:47:18 dist-test-slave-hc1k krb5kdc[26289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:47:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:47:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:47:18.916878 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:36013
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:36013,127.12.170.125:34843,127.12.170.124:33803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:47:19.024255 26302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:19.024533 26302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:19.024590 26302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:19.028574 26302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:47:19.028633 26302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:19.028657 26302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:19.028681 26302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:47:19.028708 26302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:47:19.033887 26302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:36013,127.12.170.125:34843,127.12.170.124:33803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:36013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:19.034883 26302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:19.035915 26302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:47:19.043869 26307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:19.043948 26302 server_base.cc:1034] running on GCE node
W20260926 13:47:19.043872 26308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:19.043883 26310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:19.044772 26302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:19.045751 26302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:19.046924 26302 hybrid_clock.cc:648] HybridClock initialized: now 1790430439046891 us; error 39 us; skew 500 ppm
Sep 26 13:47:19 dist-test-slave-hc1k krb5kdc[26289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430439, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:19.049547 26302 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:47:19.050413 26302 webserver.cc:460] Webserver started at http://127.12.170.126:39097/ using document root <none> and password file <none>
I20260926 13:47:19.050920 26302 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:19.050981 26302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:47:19.051162 26302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:47:19.053148 26302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "962b31582016454fae7892daf9513abb"
format_stamp: "Formatted at 2026-09-26 13:47:19 on dist-test-slave-hc1k"
server_key: "6affa6381478d6e4e43004f4a2dcae64"
server_key_iv: "e98299c40a74e21c9fff76a9694246a9"
server_key_version: "encryptionkey@0"
I20260926 13:47:19.053668 26302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "962b31582016454fae7892daf9513abb"
format_stamp: "Formatted at 2026-09-26 13:47:19 on dist-test-slave-hc1k"
server_key: "6affa6381478d6e4e43004f4a2dcae64"
server_key_iv: "e98299c40a74e21c9fff76a9694246a9"
server_key_version: "encryptionkey@0"
I20260926 13:47:19.057111 26302 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 13:47:19.059433 26317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:19.060755 26302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 13:47:19.060878 26302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "962b31582016454fae7892daf9513abb"
format_stamp: "Formatted at 2026-09-26 13:47:19 on dist-test-slave-hc1k"
server_key: "6affa6381478d6e4e43004f4a2dcae64"
server_key_iv: "e98299c40a74e21c9fff76a9694246a9"
server_key_version: "encryptionkey@0"
I20260926 13:47:19.060972 26302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:19.082163 26302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:19.085160 26302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:19.085336 26302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:19.092022 26302 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:36013
I20260926 13:47:19.092272 26369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:36013 every 8 connection(s)
I20260926 13:47:19.092974 26302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:47:19.095754 26370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:47:19.099532 26370 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36013 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:47:19.103178 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 26302
I20260926 13:47:19.103297 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:47:19.103578 12969 external_mini_cluster.cc:1428] Setting key 40d58c123e52fccece1a2ede88f6844e
Sep 26 13:47:19 dist-test-slave-hc1k krb5kdc[26289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430439, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.170.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.125@KRBTEST.COM" created.
I20260926 13:47:19.114265 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.101789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:36013 (local address 127.0.0.1:59458)
0926 13:47:19.102266 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.102318 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:19.103159 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:19.103884 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:19.103889 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:19.104320 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:19.104563 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:19.104577 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.105678 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.105680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:19.106390 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:19.106399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.106994 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.106998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:19.107102 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:19.107729 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:19.107745 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:19.109482 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:19.111482 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:19.111491 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:19.111499 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:19.111761 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:19.112146 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:19.112148 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:19.112150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:19.112258 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:19.112825 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:19.112831 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:19.112955 (+ 124us) client_negotiation.cc:769] Sending connection context
0926 13:47:19.113121 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 13:47:19.113393 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":117,"threads_started":1}
I20260926 13:47:19.114269 26373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.101789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59458 (local address 127.12.170.126:36013)
0926 13:47:19.102235 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:19.102252 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:19.102420 (+ 168us) server_negotiation.cc:407] Connection header received
0926 13:47:19.103497 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:19.103503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:19.103712 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:19.103875 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:19.104696 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.105566 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.106497 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.106851 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.106989 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:19.109637 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:19.109642 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:19.109653 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:19.109674 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:19.111370 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.111883 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.111888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.111889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.112049 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.112327 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.112330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.112331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.112694 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:19.112847 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:19.113190 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 13:47:19.113471 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":328,"spinlock_wait_cycles":16128,"thread_start_us":160,"threads_started":1}
I20260926 13:47:19.116899 26370 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.125" port: 34843 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:47:19.117606 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.117333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:34843 (local address 127.0.0.1:53314)
0926 13:47:19.117449 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.117531 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.125:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 13:47:19.117790 26319 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.170.125:34843: connect: Connection refused (error 111)
I20260926 13:47:19.118602 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.118366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:34843 (local address 127.0.0.1:53322)
0926 13:47:19.118482 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.118536 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.125:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 13:47:19.118734 26370 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.125:34843: Network error: Client connection negotiation failed: client connection to 127.12.170.125:34843: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.125.keytab.
Entry for principal kudu/127.12.170.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.125.keytab.
I20260926 13:47:19.128834 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.125:34843
--webserver_interface=127.12.170.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.125.keytab
--principal=kudu/127.12.170.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:36013,127.12.170.125:34843,127.12.170.124:33803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:47:19.167976 26370 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.125" port: 34843 } attempt: 1
I20260926 13:47:19.168900 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.168540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:34843 (local address 127.0.0.1:53328)
0926 13:47:19.168716 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.168798 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.125:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 13:47:19.169628 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.169398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:34843 (local address 127.0.0.1:53336)
0926 13:47:19.169503 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.169558 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.125:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 13:47:19.169755 26370 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.125:34843: Network error: Client connection negotiation failed: client connection to 127.12.170.125:34843: connect: Connection refused (error 111)
W20260926 13:47:19.236591 26377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:19.236861 26377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:19.236901 26377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 13:47:19.237923 26370 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.125" port: 34843 } attempt: 2
I20260926 13:47:19.238734 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.238427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:34843 (local address 127.0.0.1:53342)
0926 13:47:19.238581 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.238648 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.125:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 13:47:19.239423 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.239220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:34843 (local address 127.0.0.1:53356)
0926 13:47:19.239313 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.239360 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.125:34843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 13:47:19.239557 26370 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.125:34843: Network error: Client connection negotiation failed: client connection to 127.12.170.125:34843: connect: Connection refused (error 111)
W20260926 13:47:19.241349 26377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:47:19.241402 26377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:19.241420 26377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:19.241436 26377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:47:19.241453 26377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:47:19.246892 26377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:36013,127.12.170.125:34843,127.12.170.124:33803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.125.keytab
--principal=kudu/127.12.170.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.125:34843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.170.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:19.248006 26377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:19.249002 26377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:47:19.257323 26385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:19.257344 26383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:19.257323 26382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:19.257480 26377 server_base.cc:1034] running on GCE node
I20260926 13:47:19.258069 26377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:19.259117 26377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:19.260267 26377 hybrid_clock.cc:648] HybridClock initialized: now 1790430439260251 us; error 30 us; skew 500 ppm
Sep 26 13:47:19 dist-test-slave-hc1k krb5kdc[26289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430439, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:19.262863 26377 init.cc:377] Logged in from keytab as kudu/127.12.170.125@KRBTEST.COM (short username kudu)
I20260926 13:47:19.263738 26377 webserver.cc:460] Webserver started at http://127.12.170.125:46335/ using document root <none> and password file <none>
I20260926 13:47:19.264262 26377 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:19.264321 26377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:47:19.264487 26377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:47:19.266491 26377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-1/data/instance:
uuid: "99d65958373c4dd3801de7339941eeb9"
format_stamp: "Formatted at 2026-09-26 13:47:19 on dist-test-slave-hc1k"
server_key: "53dd8002fc7052d85c1542c5dcc150d7"
server_key_iv: "38d8e5837427f1757b99d804f4548bfa"
server_key_version: "encryptionkey@0"
I20260926 13:47:19.266989 26377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-1/wal/instance:
uuid: "99d65958373c4dd3801de7339941eeb9"
format_stamp: "Formatted at 2026-09-26 13:47:19 on dist-test-slave-hc1k"
server_key: "53dd8002fc7052d85c1542c5dcc150d7"
server_key_iv: "38d8e5837427f1757b99d804f4548bfa"
server_key_version: "encryptionkey@0"
I20260926 13:47:19.270483 26377 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 13:47:19.272750 26392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:19.273890 26377 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:47:19.273991 26377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-1/wal
uuid: "99d65958373c4dd3801de7339941eeb9"
format_stamp: "Formatted at 2026-09-26 13:47:19 on dist-test-slave-hc1k"
server_key: "53dd8002fc7052d85c1542c5dcc150d7"
server_key_iv: "38d8e5837427f1757b99d804f4548bfa"
server_key_version: "encryptionkey@0"
I20260926 13:47:19.274094 26377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:19.283349 26377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:19.286332 26377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:19.286473 26377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:19.292984 26377 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.125:34843
I20260926 13:47:19.293217 26444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.125:34843 every 8 connection(s)
I20260926 13:47:19.293921 26377 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-1/data/info.pb
I20260926 13:47:19.294714 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 26377
I20260926 13:47:19.294814 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-1/wal/instance
I20260926 13:47:19.295085 12969 external_mini_cluster.cc:1428] Setting key 79f7aa28d65a78f2763f68eff6eb7afd
I20260926 13:47:19.296802 26445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:47:19.300573 26445 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36013 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.170.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.124@KRBTEST.COM" created.
Sep 26 13:47:19 dist-test-slave-hc1k krb5kdc[26289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430439, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.125@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:47:19.313079 26373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.302349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59464 (local address 127.12.170.126:36013)
0926 13:47:19.302483 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:19.302487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:19.303311 (+ 824us) server_negotiation.cc:407] Connection header received
0926 13:47:19.304272 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:19.304275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:19.304324 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:19.304413 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:19.305520 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.306039 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.307121 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.307350 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.307436 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:19.309862 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:19.309865 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:19.309867 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:19.309892 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:19.311236 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.311732 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.311735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.311736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.311806 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.312098 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.312100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.312101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.312268 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:19.312337 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:19.312833 (+ 496us) server_negotiation.cc:299] Negotiation successful
0926 13:47:19.312934 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 13:47:19.313885 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.302751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:36013 (local address 127.0.0.1:59464)
0926 13:47:19.303176 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.303216 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:19.304060 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:19.304559 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:19.304566 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:19.304892 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:19.305376 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:19.305389 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.306156 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.306158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:19.307008 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:19.307017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.307483 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.307487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:19.307678 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:19.308201 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:19.308217 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:19.309691 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:19.311337 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:19.311342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:19.311353 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:19.311631 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:19.311872 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:19.311874 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:19.311876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:19.312037 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:19.312346 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:19.312351 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:19.312605 (+ 254us) client_negotiation.cc:769] Sending connection context
0926 13:47:19.312752 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 13:47:19.313006 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":101,"threads_started":1}
I20260926 13:47:19.315212 26445 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.125" port: 34843 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.124.keytab.
Entry for principal kudu/127.12.170.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.124.keytab.
I20260926 13:47:19.318406 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.124:33803
--webserver_interface=127.12.170.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.124.keytab
--principal=kudu/127.12.170.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:36013,127.12.170.125:34843,127.12.170.124:33803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:47:19 dist-test-slave-hc1k krb5kdc[26289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430439, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.125@KRBTEST.COM for kudu/127.12.170.125@KRBTEST.COM
I20260926 13:47:19.323843 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.315690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:34843 (local address 127.0.0.1:53362)
0926 13:47:19.315791 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.315803 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:19.315866 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:19.316411 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:19.316414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:19.316539 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:19.316642 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:19.316648 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.317594 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.317596 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:19.318162 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:19.318171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.318865 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.318868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:19.318957 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:19.319391 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:19.319401 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:19.320623 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:19.322303 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:19.322307 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:19.322309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:19.322537 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:19.322786 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:19.322789 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:19.322791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:19.322875 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:19.323287 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:19.323290 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:19.323383 (+ 93us) client_negotiation.cc:769] Sending connection context
0926 13:47:19.323495 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 13:47:19.323615 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":1,"spinlock_wait_cycles":1408}
I20260926 13:47:19.323861 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.315775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53362 (local address 127.12.170.125:34843)
0926 13:47:19.316068 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:19.316079 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:19.316094 (+ 15us) server_negotiation.cc:407] Connection header received
0926 13:47:19.316129 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:19.316134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:19.316301 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:19.316398 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:19.316737 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.317504 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.318268 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.318766 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.318868 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:19.320741 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:19.320749 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:19.320760 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:19.320779 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:19.322199 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.322631 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.322636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.322637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.322709 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.322960 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.322964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.322965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.323199 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:19.323294 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:19.323559 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 13:47:19.323730 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":224,"spinlock_wait_cycles":3968,"thread_start_us":144,"threads_started":1}
I20260926 13:47:19.325496 26445 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:47:19.326146 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.325877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:33803 (local address 127.0.0.1:52992)
0926 13:47:19.325991 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.326081 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 13:47:19.326325 26395 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.170.124:33803: connect: Connection refused (error 111)
I20260926 13:47:19.327100 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.326880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:33803 (local address 127.0.0.1:53004)
0926 13:47:19.326983 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.327043 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 13:47:19.327245 26445 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.124:33803: Network error: Client connection negotiation failed: client connection to 127.12.170.124:33803: connect: Connection refused (error 111)
I20260926 13:47:19.330716 26370 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.125" port: 34843 } attempt: 3
Sep 26 13:47:19 dist-test-slave-hc1k krb5kdc[26289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430439, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for kudu/127.12.170.125@KRBTEST.COM
I20260926 13:47:19.338398 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.331172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53378 (local address 127.12.170.125:34843)
0926 13:47:19.331266 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:19.331269 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:19.331282 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:47:19.331410 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:19.331412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:19.331445 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:19.331503 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:19.331973 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.332486 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.333495 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.333707 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.333770 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:19.335754 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:19.335757 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:19.335759 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:19.335779 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:19.337024 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.337435 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.337437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.337438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.337501 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.337730 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.337732 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.337733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.337883 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:19.337948 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:19.338200 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 13:47:19.338286 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 13:47:19.338444 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.331100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:34843 (local address 127.0.0.1:53378)
0926 13:47:19.331233 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.331247 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:19.331333 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:19.331532 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:19.331535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:19.331745 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:19.331865 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:19.331872 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.332572 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.332575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:19.333399 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:19.333411 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.333788 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.333792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:19.333892 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:19.334323 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:19.334336 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:19.335593 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:19.337120 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:19.337124 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:19.337126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:19.337357 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:19.337568 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:19.337571 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:19.337573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:19.337657 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:19.337956 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:19.337958 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:19.338058 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 13:47:19.338162 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 13:47:19.338290 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 13:47:19.339682 26370 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:47:19.340301 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.340111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:33803 (local address 127.0.0.1:53020)
0926 13:47:19.340199 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.340243 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 13:47:19.340843 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.340655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:33803 (local address 127.0.0.1:53026)
0926 13:47:19.340760 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.340797 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 13:47:19.340965 26370 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.124:33803: Network error: Client connection negotiation failed: client connection to 127.12.170.124:33803: connect: Connection refused (error 111)
I20260926 13:47:19.372154 26370 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } attempt: 1
I20260926 13:47:19.373001 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.372716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:33803 (local address 127.0.0.1:53036)
0926 13:47:19.372854 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.372917 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 13:47:19.373659 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.373447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:33803 (local address 127.0.0.1:53048)
0926 13:47:19.373555 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.373600 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 13:47:19.373812 26370 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.124:33803: Network error: Client connection negotiation failed: client connection to 127.12.170.124:33803: connect: Connection refused (error 111)
I20260926 13:47:19.376480 26445 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } attempt: 1
I20260926 13:47:19.377193 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.376889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:33803 (local address 127.0.0.1:53056)
0926 13:47:19.377035 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.377111 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 13:47:19.377888 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.377675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:33803 (local address 127.0.0.1:53064)
0926 13:47:19.377772 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.377826 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 13:47:19.378113 26445 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.124:33803: Network error: Client connection negotiation failed: client connection to 127.12.170.124:33803: connect: Connection refused (error 111)
W20260926 13:47:19.428972 26451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:19.429210 26451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:19.429247 26451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:19.433049 26451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:47:19.433096 26451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:19.433113 26451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:19.433128 26451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:47:19.433141 26451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:47:19.438149 26451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:36013,127.12.170.125:34843,127.12.170.124:33803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.124.keytab
--principal=kudu/127.12.170.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.124:33803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.170.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:19.439079 26451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:19.440033 26451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:47:19.446255 26445 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } attempt: 2
I20260926 13:47:19.447068 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.446720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:33803 (local address 127.0.0.1:53074)
0926 13:47:19.446881 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.446936 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260926 13:47:19.447860 26457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:19.447975 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.447738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:33803 (local address 127.0.0.1:53084)
0926 13:47:19.447864 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.447913 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 13:47:19.447962 26460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:19.447873 26458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:19.448128 26445 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.124:33803: Network error: Client connection negotiation failed: client connection to 127.12.170.124:33803: connect: Connection refused (error 111)
I20260926 13:47:19.448241 26451 server_base.cc:1034] running on GCE node
I20260926 13:47:19.448747 26451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:19.448973 26370 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } attempt: 2
I20260926 13:47:19.449597 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.449319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:33803 (local address 127.0.0.1:53086)
0926 13:47:19.449451 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.449520 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 13:47:19.449895 26451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:19.450358 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.450039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:33803 (local address 127.0.0.1:53094)
0926 13:47:19.450254 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.450305 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260926 13:47:19.450477 26370 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.124:33803: Network error: Client connection negotiation failed: client connection to 127.12.170.124:33803: connect: Connection refused (error 111)
I20260926 13:47:19.451099 26451 hybrid_clock.cc:648] HybridClock initialized: now 1790430439451075 us; error 32 us; skew 500 ppm
Sep 26 13:47:19 dist-test-slave-hc1k krb5kdc[26289](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430439, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:19.453786 26451 init.cc:377] Logged in from keytab as kudu/127.12.170.124@KRBTEST.COM (short username kudu)
I20260926 13:47:19.454643 26451 webserver.cc:460] Webserver started at http://127.12.170.124:39187/ using document root <none> and password file <none>
I20260926 13:47:19.455134 26451 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:19.455188 26451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:47:19.455343 26451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:47:19.457295 26451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-2/data/instance:
uuid: "2abd7a60d99645709bcba01b883f0c4d"
format_stamp: "Formatted at 2026-09-26 13:47:19 on dist-test-slave-hc1k"
server_key: "31860a7082644066981f6444bf3b3e39"
server_key_iv: "89f45543b834b8cee70b27c15a5d26db"
server_key_version: "encryptionkey@0"
I20260926 13:47:19.457775 26451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-2/wal/instance:
uuid: "2abd7a60d99645709bcba01b883f0c4d"
format_stamp: "Formatted at 2026-09-26 13:47:19 on dist-test-slave-hc1k"
server_key: "31860a7082644066981f6444bf3b3e39"
server_key_iv: "89f45543b834b8cee70b27c15a5d26db"
server_key_version: "encryptionkey@0"
I20260926 13:47:19.461295 26451 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260926 13:47:19.463524 26467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:19.464820 26451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 13:47:19.464920 26451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-2/wal
uuid: "2abd7a60d99645709bcba01b883f0c4d"
format_stamp: "Formatted at 2026-09-26 13:47:19 on dist-test-slave-hc1k"
server_key: "31860a7082644066981f6444bf3b3e39"
server_key_iv: "89f45543b834b8cee70b27c15a5d26db"
server_key_version: "encryptionkey@0"
I20260926 13:47:19.465029 26451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:19.503690 26451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:19.506572 26451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:19.506718 26451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:19.513489 26451 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.124:33803
I20260926 13:47:19.513514 26519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.124:33803 every 8 connection(s)
I20260926 13:47:19.514408 26451 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-2/data/info.pb
I20260926 13:47:19.514781 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 26451
I20260926 13:47:19.514897 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790430291939677-12969-0/minicluster-data/master-2/wal/instance
I20260926 13:47:19.515202 12969 external_mini_cluster.cc:1428] Setting key 1bac205aa84e6a4cb2354e6e95111413
I20260926 13:47:19.517477 26520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:47:19.521068 26520 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36013 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:47:19 dist-test-slave-hc1k krb5kdc[26289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.124@KRBTEST.COM
Sep 26 13:47:19 dist-test-slave-hc1k krb5kdc[26289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430439, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.124@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:47:19.528239 26523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.516561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53100 (local address 127.12.170.124:33803)
0926 13:47:19.516971 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:19.516986 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:19.517021 (+ 35us) server_negotiation.cc:407] Connection header received
0926 13:47:19.517681 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:19.517698 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:19.518018 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:19.518322 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:19.519095 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.520124 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.520938 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.521422 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.521560 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:19.524012 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:19.524017 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:19.524025 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:19.524052 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:19.525520 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.525911 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.525917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.525921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.526026 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.526273 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.526277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.526279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.526638 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:19.526842 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:19.527056 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 13:47:19.527334 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":135,"threads_started":1}
I20260926 13:47:19.531250 26524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.522603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:36013 (local address 127.0.0.1:59480)
0926 13:47:19.522975 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.523021 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:19.523139 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:19.523403 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:19.523407 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:19.523786 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:19.523910 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:19.523916 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.524635 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.524637 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:19.525232 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:19.525240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.525645 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.525648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:19.525750 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:19.526196 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:19.526207 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:19.527609 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:19.529701 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:19.529709 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:19.529714 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:19.529952 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:19.530236 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:19.530239 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:19.530241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:19.530331 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:19.530728 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:19.530734 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:19.530855 (+ 121us) client_negotiation.cc:769] Sending connection context
0926 13:47:19.530984 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 13:47:19.531118 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":108,"threads_started":1}
I20260926 13:47:19.531347 26373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.522698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59480 (local address 127.12.170.126:36013)
0926 13:47:19.522833 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:19.522838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:19.523103 (+ 265us) server_negotiation.cc:407] Connection header received
0926 13:47:19.523218 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:19.523221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:19.523273 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:19.523355 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:19.524024 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.524522 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.525337 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.525553 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.525638 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:19.527775 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:19.527778 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:19.527780 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:19.527805 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:19.529603 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.530043 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.530047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.530049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.530147 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.530433 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.530437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.530439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.530632 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:19.530708 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:19.531091 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 13:47:19.531208 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 13:47:19.532349 26520 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.125" port: 34843 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:47:19 dist-test-slave-hc1k krb5kdc[26289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
Sep 26 13:47:19 dist-test-slave-hc1k krb5kdc[26289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430439, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.124@KRBTEST.COM for kudu/127.12.170.125@KRBTEST.COM
I20260926 13:47:19.539126 26525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.530320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59492 (local address 127.12.170.126:36013)
0926 13:47:19.530637 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:19.530644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:19.530654 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:47:19.530683 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:19.530686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:19.530726 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:19.530806 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:19.531490 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.532157 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.532984 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.533312 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.533382 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:19.535920 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:19.535923 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:19.535925 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:19.535945 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:19.537687 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.538046 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.538049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.538051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.538122 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.538353 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.538356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.538358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.538520 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:19.538725 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:19.538854 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 13:47:19.538958 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":99,"threads_started":1}
I20260926 13:47:19.539458 26445 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } attempt: 3
I20260926 13:47:19.540126 26524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.532769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:34843 (local address 127.0.0.1:53382)
0926 13:47:19.532881 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.532895 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:19.532966 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:19.533170 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:19.533172 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:19.533289 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:19.533377 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:19.533383 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.534096 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.534099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:19.534888 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:19.534897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.535424 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.535428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:19.535526 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:19.535909 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:19.535922 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:19.537105 (+ 1183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:19.538607 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:19.538610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:19.538612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:19.538826 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:19.539114 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:19.539117 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:19.539119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:19.539210 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:19.539670 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:19.539673 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:19.539766 (+ 93us) client_negotiation.cc:769] Sending connection context
0926 13:47:19.539860 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 13:47:19.539981 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 13:47:19.540161 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.532871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53382 (local address 127.12.170.125:34843)
0926 13:47:19.532970 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:19.532974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:19.532987 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:47:19.533030 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:19.533033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:19.533080 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:19.533149 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:19.533492 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.533963 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.534995 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.535324 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.535429 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:19.537230 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:19.537233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:19.537235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:19.537260 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:19.538511 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.538922 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.538926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.538928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.539013 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.539299 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.539302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.539303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.539509 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:19.539607 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:19.539917 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 13:47:19.540021 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 13:47:19.540858 26520 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:47:19 dist-test-slave-hc1k krb5kdc[26289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430439, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.125@KRBTEST.COM for kudu/127.12.170.124@KRBTEST.COM
Sep 26 13:47:19 dist-test-slave-hc1k krb5kdc[26289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.125@KRBTEST.COM
Sep 26 13:47:19 dist-test-slave-hc1k krb5kdc[26289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430439, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.124@KRBTEST.COM for kudu/127.12.170.124@KRBTEST.COM
I20260926 13:47:19.549624 26370 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } attempt: 3
I20260926 13:47:19.549721 26523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.540303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53110 (local address 127.12.170.124:33803)
0926 13:47:19.540513 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:19.540517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:19.540528 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:47:19.540558 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:19.540560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:19.540598 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:19.540674 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:19.541079 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.541826 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.543129 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.543363 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.543481 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:19.546381 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:19.546385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:19.546388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:19.546411 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:19.548098 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.548562 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.548565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.548567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.548658 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.548947 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.548950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.548952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.549122 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:19.549205 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:19.549447 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 13:47:19.549561 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":145}
I20260926 13:47:19.549721 26448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.539850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:33803 (local address 127.0.0.1:53110)
0926 13:47:19.540082 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.540096 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:19.540174 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:19.540708 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:19.540711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:19.540873 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:19.540957 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:19.540963 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.542561 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.542563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:19.543036 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:19.543041 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.543930 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.543933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:19.544002 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:19.544433 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:19.544447 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:19.546244 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:19.548212 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:19.548216 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:19.548218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:19.548436 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:19.548738 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:19.548741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:19.548743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:19.548840 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:19.549214 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:19.549217 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:19.549306 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 13:47:19.549389 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 13:47:19.549561 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260926 13:47:19.551478 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.540403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53386 (local address 127.12.170.125:34843)
0926 13:47:19.540810 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:19.540814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:19.540824 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:47:19.540853 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:19.540855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:19.540890 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:19.540961 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:19.541916 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.542468 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.543476 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.543757 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.543837 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:19.547636 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:19.547640 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:19.547642 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:19.547697 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:19.549557 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.550001 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.550005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.550006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.550084 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.550376 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.550379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.550380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.550568 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:19.550780 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:19.551132 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 13:47:19.551337 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347}
I20260926 13:47:19.552011 26524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.541281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:33803 (local address 127.0.0.1:53116)
0926 13:47:19.541366 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.541378 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:19.541437 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:19.541818 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:19.541821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:19.541967 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:19.542036 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:19.542054 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.543416 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.543419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:19.544032 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:19.544040 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.544755 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.544759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:19.544864 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:19.545260 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:19.545278 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:19.547465 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:19.549513 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:19.549516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:19.549518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:19.550820 (+ 1302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:19.551071 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:19.551074 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:19.551076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:19.551164 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:19.551577 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:19.551580 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:19.551684 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 13:47:19.551761 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 13:47:19.551869 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":43}
I20260926 13:47:19.552824 26526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.541332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53116 (local address 127.12.170.124:33803)
0926 13:47:19.541640 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:19.541645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:19.541659 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:47:19.541686 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:19.541688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:19.541724 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:19.541795 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:19.542153 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.543305 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.544428 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.544651 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.544748 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:19.547594 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:19.547598 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:19.547600 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:19.547620 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:19.549403 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.550909 (+ 1506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.550912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.550913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.550986 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.551255 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.551257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.551259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.551481 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:19.552504 (+ 1023us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:19.552575 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 13:47:19.552675 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":258,"thread_start_us":110,"threads_started":1}
I20260926 13:47:19.553750 26445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99d65958373c4dd3801de7339941eeb9: Bootstrap starting.
I20260926 13:47:19.554975 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:47:19.555603 26520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2abd7a60d99645709bcba01b883f0c4d: Bootstrap starting.
I20260926 13:47:19.556903 26445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99d65958373c4dd3801de7339941eeb9: Neither blocks nor log segments found. Creating new log.
Sep 26 13:47:19 dist-test-slave-hc1k krb5kdc[26289](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430439, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for kudu/127.12.170.124@KRBTEST.COM
I20260926 13:47:19.558079 26520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2abd7a60d99645709bcba01b883f0c4d: Neither blocks nor log segments found. Creating new log.
I20260926 13:47:19.558126 26445 log.cc:824] T 00000000000000000000000000000000 P 99d65958373c4dd3801de7339941eeb9: Log is configured to *not* fsync() on all Append() calls
I20260926 13:47:19.558940 26520 log.cc:824] T 00000000000000000000000000000000 P 2abd7a60d99645709bcba01b883f0c4d: Log is configured to *not* fsync() on all Append() calls
I20260926 13:47:19.561059 26520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2abd7a60d99645709bcba01b883f0c4d: No bootstrap required, opened a new log
I20260926 13:47:19.561352 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.550092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:33803 (local address 127.0.0.1:53128)
0926 13:47:19.550200 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:19.550214 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:19.550289 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:19.550592 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:19.550595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:19.550746 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:19.550827 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:19.550832 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.551854 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.551858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:19.552599 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:19.552611 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.555959 (+ 3348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.555963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:19.556051 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:19.556505 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:19.556520 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:19.557963 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:19.559878 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:19.559882 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:19.559884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:19.560114 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:19.560410 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:19.560413 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:19.560415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:19.560500 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:19.560895 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:19.560898 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:19.560983 (+ 85us) client_negotiation.cc:769] Sending connection context
0926 13:47:19.561066 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 13:47:19.561195 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 13:47:19.561901 26523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.550327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53128 (local address 127.12.170.124:33803)
0926 13:47:19.550410 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:19.550414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:19.550424 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:47:19.550452 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:19.550454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:19.550490 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:19.550557 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:19.550923 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.551735 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.553009 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.555828 (+ 2819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.556275 (+ 447us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:19.558110 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:19.558113 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:19.558116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:19.558140 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:19.559746 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.560233 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.560236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.560238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.560316 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.560596 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.560599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.560601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.560797 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:19.561590 (+ 793us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:19.561668 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 13:47:19.561767 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 13:47:19.565402 26520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2abd7a60d99645709bcba01b883f0c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962b31582016454fae7892daf9513abb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36013 } } peers { permanent_uuid: "99d65958373c4dd3801de7339941eeb9" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 34843 } } peers { permanent_uuid: "2abd7a60d99645709bcba01b883f0c4d" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } }
I20260926 13:47:19.565307 26370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb: Bootstrap starting.
I20260926 13:47:19.565752 26520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2abd7a60d99645709bcba01b883f0c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:47:19.565816 26520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2abd7a60d99645709bcba01b883f0c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2abd7a60d99645709bcba01b883f0c4d, State: Initialized, Role: FOLLOWER
I20260926 13:47:19.566298 26520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2abd7a60d99645709bcba01b883f0c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962b31582016454fae7892daf9513abb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36013 } } peers { permanent_uuid: "99d65958373c4dd3801de7339941eeb9" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 34843 } } peers { permanent_uuid: "2abd7a60d99645709bcba01b883f0c4d" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } }
I20260926 13:47:19.567615 26520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2abd7a60d99645709bcba01b883f0c4d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:47:19.567790 26450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.558177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53388 (local address 127.12.170.125:34843)
0926 13:47:19.558281 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:19.558285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:19.559092 (+ 807us) server_negotiation.cc:407] Connection header received
0926 13:47:19.559273 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:19.559276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:19.559324 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:19.559370 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:19.560302 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.560986 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.562023 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.562331 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.562397 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:19.564600 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:19.564603 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:19.564606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:19.564630 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:19.566142 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.566577 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.566580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.566582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.566652 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.566933 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.566936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.566938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.567126 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:19.567240 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:19.567487 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 13:47:19.567594 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:47:19.568861 26370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb: Neither blocks nor log segments found. Creating new log.
I20260926 13:47:19.569875 26370 log.cc:824] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb: Log is configured to *not* fsync() on all Append() calls
I20260926 13:47:19.570752 26526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.557796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53142 (local address 127.12.170.124:33803)
0926 13:47:19.557917 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:19.557920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:19.559070 (+ 1150us) server_negotiation.cc:407] Connection header received
0926 13:47:19.562145 (+ 3075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:19.562148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:19.562186 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:19.562228 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:19.563022 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.563501 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.564587 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.564945 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.565014 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:19.566537 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:19.566541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:19.566543 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:19.566567 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:19.568358 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.568789 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.568793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.568794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.568872 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.569163 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.569166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.569167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.569353 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:19.569846 (+ 493us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:19.570181 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 13:47:19.570616 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 13:47:19.572059 26445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99d65958373c4dd3801de7339941eeb9: No bootstrap required, opened a new log
I20260926 13:47:19.571746 26451 master_runner.cc:186] Error getting master registration for 127.12.170.126:36013: OK, instance_id { permanent_uuid: "962b31582016454fae7892daf9513abb" instance_seqno: 1790430439090161 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 13:47:19.574669 26451 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 13:47:19.574937 26370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb: No bootstrap required, opened a new log
I20260926 13:47:19.576004 26445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99d65958373c4dd3801de7339941eeb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962b31582016454fae7892daf9513abb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36013 } } peers { permanent_uuid: "99d65958373c4dd3801de7339941eeb9" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 34843 } } peers { permanent_uuid: "2abd7a60d99645709bcba01b883f0c4d" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } }
I20260926 13:47:19.576180 26525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:19.557193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59504 (local address 127.12.170.126:36013)
0926 13:47:19.557304 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:19.557308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:19.558742 (+ 1434us) server_negotiation.cc:407] Connection header received
0926 13:47:19.559246 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:19.559249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:19.559301 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:19.559377 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:19.560323 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.561037 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.570613 (+ 9576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:19.570950 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:19.571022 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:19.572815 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:19.572819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:19.572822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:19.572846 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:19.574332 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.574819 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.574823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.574824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.574908 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:19.575188 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:19.575191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:19.575193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:19.575393 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:19.575510 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:19.575923 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 13:47:19.576042 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 13:47:19.576217 26445 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99d65958373c4dd3801de7339941eeb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:47:19.576267 26445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99d65958373c4dd3801de7339941eeb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99d65958373c4dd3801de7339941eeb9, State: Initialized, Role: FOLLOWER
I20260926 13:47:19.576678 26536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2abd7a60d99645709bcba01b883f0c4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962b31582016454fae7892daf9513abb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36013 } } peers { permanent_uuid: "99d65958373c4dd3801de7339941eeb9" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 34843 } } peers { permanent_uuid: "2abd7a60d99645709bcba01b883f0c4d" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } } }
I20260926 13:47:19.576853 26536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2abd7a60d99645709bcba01b883f0c4d [sys.catalog]: This master's current role is: FOLLOWER
W20260926 13:47:19.577277 26547 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2abd7a60d99645709bcba01b883f0c4d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 13:47:19.577221 26445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99d65958373c4dd3801de7339941eeb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962b31582016454fae7892daf9513abb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36013 } } peers { permanent_uuid: "99d65958373c4dd3801de7339941eeb9" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 34843 } } peers { permanent_uuid: "2abd7a60d99645709bcba01b883f0c4d" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } }
W20260926 13:47:19.577343 26547 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:47:19.577731 26550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99d65958373c4dd3801de7339941eeb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962b31582016454fae7892daf9513abb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36013 } } peers { permanent_uuid: "99d65958373c4dd3801de7339941eeb9" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 34843 } } peers { permanent_uuid: "2abd7a60d99645709bcba01b883f0c4d" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } } }
I20260926 13:47:19.577836 26550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99d65958373c4dd3801de7339941eeb9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:47:19.578653 26445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99d65958373c4dd3801de7339941eeb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:47:19.579119 26370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962b31582016454fae7892daf9513abb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36013 } } peers { permanent_uuid: "99d65958373c4dd3801de7339941eeb9" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 34843 } } peers { permanent_uuid: "2abd7a60d99645709bcba01b883f0c4d" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } }
I20260926 13:47:19.579317 26370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:47:19.579370 26370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 962b31582016454fae7892daf9513abb, State: Initialized, Role: FOLLOWER
I20260926 13:47:19.579849 26370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962b31582016454fae7892daf9513abb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36013 } } peers { permanent_uuid: "99d65958373c4dd3801de7339941eeb9" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 34843 } } peers { permanent_uuid: "2abd7a60d99645709bcba01b883f0c4d" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } }
I20260926 13:47:19.581070 26370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:47:19.581717 26377 master_runner.cc:186] Error getting master registration for 127.12.170.126:36013: OK, instance_id { permanent_uuid: "962b31582016454fae7892daf9513abb" instance_seqno: 1790430439090161 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 13:47:19.585470 26377 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 13:47:19.586664 26573 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:47:19.586735 26573 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 13:47:19.586822 26323 tablet.cc:2369] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:47:19.586925 26557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962b31582016454fae7892daf9513abb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36013 } } peers { permanent_uuid: "99d65958373c4dd3801de7339941eeb9" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 34843 } } peers { permanent_uuid: "2abd7a60d99645709bcba01b883f0c4d" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } } }
I20260926 13:47:19.587031 26557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:47:19.590320 26302 master_runner.cc:263] No leader master found from master 962b31582016454fae7892daf9513abb
I20260926 13:47:19.590381 26302 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 13:47:19.594864 26562 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 99d65958373c4dd3801de7339941eeb9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:47:19.595008 26562 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:47:19.657673 26557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:47:19.657883 26557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962b31582016454fae7892daf9513abb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36013 } } peers { permanent_uuid: "99d65958373c4dd3801de7339941eeb9" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 34843 } } peers { permanent_uuid: "2abd7a60d99645709bcba01b883f0c4d" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } }
I20260926 13:47:19.658950 26557 leader_election.cc:288] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99d65958373c4dd3801de7339941eeb9 (127.12.170.125:34843), 2abd7a60d99645709bcba01b883f0c4d (127.12.170.124:33803)
I20260926 13:47:19.659606 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "962b31582016454fae7892daf9513abb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2abd7a60d99645709bcba01b883f0c4d" is_pre_election: true
I20260926 13:47:19.659974 26494 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2abd7a60d99645709bcba01b883f0c4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 962b31582016454fae7892daf9513abb in term 0.
I20260926 13:47:19.660394 26320 leader_election.cc:302] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2abd7a60d99645709bcba01b883f0c4d, 962b31582016454fae7892daf9513abb; no voters:
I20260926 13:47:19.661146 26419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "962b31582016454fae7892daf9513abb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d65958373c4dd3801de7339941eeb9" is_pre_election: true
I20260926 13:47:19.661430 26419 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 99d65958373c4dd3801de7339941eeb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 962b31582016454fae7892daf9513abb in term 0.
I20260926 13:47:19.662000 26557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:47:19.662089 26557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:47:19.662141 26557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:47:19.663432 26557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962b31582016454fae7892daf9513abb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36013 } } peers { permanent_uuid: "99d65958373c4dd3801de7339941eeb9" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 34843 } } peers { permanent_uuid: "2abd7a60d99645709bcba01b883f0c4d" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } }
I20260926 13:47:19.663851 26557 leader_election.cc:288] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [CANDIDATE]: Term 1 election: Requested vote from peers 99d65958373c4dd3801de7339941eeb9 (127.12.170.125:34843), 2abd7a60d99645709bcba01b883f0c4d (127.12.170.124:33803)
I20260926 13:47:19.664288 26494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "962b31582016454fae7892daf9513abb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2abd7a60d99645709bcba01b883f0c4d"
I20260926 13:47:19.664415 26494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2abd7a60d99645709bcba01b883f0c4d [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:47:19.665721 26494 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2abd7a60d99645709bcba01b883f0c4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 962b31582016454fae7892daf9513abb in term 1.
I20260926 13:47:19.666050 26320 leader_election.cc:302] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2abd7a60d99645709bcba01b883f0c4d, 962b31582016454fae7892daf9513abb; no voters:
I20260926 13:47:19.666471 26419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "962b31582016454fae7892daf9513abb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99d65958373c4dd3801de7339941eeb9"
I20260926 13:47:19.666610 26419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99d65958373c4dd3801de7339941eeb9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:47:19.667831 26419 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 99d65958373c4dd3801de7339941eeb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 962b31582016454fae7892daf9513abb in term 1.
I20260926 13:47:19.668262 26557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:47:19.668490 26557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [term 1 LEADER]: Becoming Leader. State: Replica: 962b31582016454fae7892daf9513abb, State: Running, Role: LEADER
I20260926 13:47:19.670773 26557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962b31582016454fae7892daf9513abb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36013 } } peers { permanent_uuid: "99d65958373c4dd3801de7339941eeb9" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 34843 } } peers { permanent_uuid: "2abd7a60d99645709bcba01b883f0c4d" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } }
I20260926 13:47:19.673700 26557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 962b31582016454fae7892daf9513abb. Latest consensus state: current_term: 1 leader_uuid: "962b31582016454fae7892daf9513abb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962b31582016454fae7892daf9513abb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36013 } } peers { permanent_uuid: "99d65958373c4dd3801de7339941eeb9" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 34843 } } peers { permanent_uuid: "2abd7a60d99645709bcba01b883f0c4d" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } } }
I20260926 13:47:19.673818 26557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [sys.catalog]: This master's current role is: LEADER
I20260926 13:47:19.675535 26578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:47:19.676496 26578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:47:19.682011 26494 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2abd7a60d99645709bcba01b883f0c4d [term 1 FOLLOWER]: Refusing update from remote peer 962b31582016454fae7892daf9513abb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:47:19.682696 26576 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2abd7a60d99645709bcba01b883f0c4d" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:47:19.686471 26536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2abd7a60d99645709bcba01b883f0c4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 962b31582016454fae7892daf9513abb. Latest consensus state: current_term: 1 leader_uuid: "962b31582016454fae7892daf9513abb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962b31582016454fae7892daf9513abb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36013 } } peers { permanent_uuid: "99d65958373c4dd3801de7339941eeb9" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 34843 } } peers { permanent_uuid: "2abd7a60d99645709bcba01b883f0c4d" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } } }
I20260926 13:47:19.686619 26536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2abd7a60d99645709bcba01b883f0c4d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:47:19.690032 26419 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 99d65958373c4dd3801de7339941eeb9 [term 1 FOLLOWER]: Refusing update from remote peer 962b31582016454fae7892daf9513abb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:47:19.690613 26576 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [LEADER]: Connected to new peer: Peer: permanent_uuid: "99d65958373c4dd3801de7339941eeb9" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 34843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:47:19.694923 26550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99d65958373c4dd3801de7339941eeb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 962b31582016454fae7892daf9513abb. Latest consensus state: current_term: 1 leader_uuid: "962b31582016454fae7892daf9513abb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962b31582016454fae7892daf9513abb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36013 } } peers { permanent_uuid: "99d65958373c4dd3801de7339941eeb9" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 34843 } } peers { permanent_uuid: "2abd7a60d99645709bcba01b883f0c4d" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } } }
I20260926 13:47:19.695065 26550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99d65958373c4dd3801de7339941eeb9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:47:19.702689 26576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "962b31582016454fae7892daf9513abb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962b31582016454fae7892daf9513abb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36013 } } peers { permanent_uuid: "99d65958373c4dd3801de7339941eeb9" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 34843 } } peers { permanent_uuid: "2abd7a60d99645709bcba01b883f0c4d" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } } }
I20260926 13:47:19.702876 26576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [sys.catalog]: This master's current role is: LEADER
I20260926 13:47:19.704389 26536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2abd7a60d99645709bcba01b883f0c4d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "962b31582016454fae7892daf9513abb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962b31582016454fae7892daf9513abb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36013 } } peers { permanent_uuid: "99d65958373c4dd3801de7339941eeb9" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 34843 } } peers { permanent_uuid: "2abd7a60d99645709bcba01b883f0c4d" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } } }
I20260926 13:47:19.704536 26536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2abd7a60d99645709bcba01b883f0c4d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:47:19.707592 26579 mvcc.cc:205] Tried to move back new op lower bound from 7333603080929845248 to 7333603080895655936. Current Snapshot: MvccSnapshot[applied={T|T < 7333603080929845248}]
I20260926 13:47:19.708704 26557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "962b31582016454fae7892daf9513abb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962b31582016454fae7892daf9513abb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36013 } } peers { permanent_uuid: "99d65958373c4dd3801de7339941eeb9" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 34843 } } peers { permanent_uuid: "2abd7a60d99645709bcba01b883f0c4d" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } } }
I20260926 13:47:19.708832 26557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb [sys.catalog]: This master's current role is: LEADER
I20260926 13:47:19.710026 26550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99d65958373c4dd3801de7339941eeb9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "962b31582016454fae7892daf9513abb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "962b31582016454fae7892daf9513abb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36013 } } peers { permanent_uuid: "99d65958373c4dd3801de7339941eeb9" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 34843 } } peers { permanent_uuid: "2abd7a60d99645709bcba01b883f0c4d" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 33803 } } }
I20260926 13:47:19.710155 26550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99d65958373c4dd3801de7339941eeb9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:47:19.710894 26582 mvcc.cc:205] Tried to move back new op lower bound from 7333603080929845248 to 7333603080895655936. Current Snapshot: MvccSnapshot[applied={T|T < 7333603080929845248}]
I20260926 13:47:19.711169 26580 mvcc.cc:205] Tried to move back new op lower bound from 7333603080929845248 to 7333603080895655936. Current Snapshot: MvccSnapshot[applied={T|T < 7333603080929845248}]
I20260926 13:47:19.716451 26578 catalog_manager.cc:1347] Generated new cluster ID: b385926314f34f1ca072fc92eca9f862
I20260926 13:47:19.716542 26578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:47:19.752363 26578 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:47:19.753553 26578 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:47:19.767316 26578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 962b31582016454fae7892daf9513abb: Generated new TSK 0
I20260926 13:47:19.768165 26578 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:47:20.578701 26547 catalog_manager.cc:1259] Loaded cluster ID: b385926314f34f1ca072fc92eca9f862
I20260926 13:47:20.578780 26547 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2abd7a60d99645709bcba01b883f0c4d: loading cluster ID for follower catalog manager: success
I20260926 13:47:20.580513 26547 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2abd7a60d99645709bcba01b883f0c4d: acquiring CA information for follower catalog manager: success
I20260926 13:47:20.581718 26547 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2abd7a60d99645709bcba01b883f0c4d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:47:20.596338 26562 catalog_manager.cc:1259] Loaded cluster ID: b385926314f34f1ca072fc92eca9f862
I20260926 13:47:20.596414 26562 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 99d65958373c4dd3801de7339941eeb9: loading cluster ID for follower catalog manager: success
I20260926 13:47:20.597956 26562 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 99d65958373c4dd3801de7339941eeb9: acquiring CA information for follower catalog manager: success
I20260926 13:47:20.599054 26562 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 99d65958373c4dd3801de7339941eeb9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:47:20.837728 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 26302
I20260926 13:47:20.843061 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 26377
I20260926 13:47:20.847896 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 26451
2026-09-26T13:47:20Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3008 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:47:20 dist-test-slave-hc1k krb5kdc[26593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:47:20 dist-test-slave-hc1k krb5kdc[26593](info): set up 2 sockets
Sep 26 13:47:20 dist-test-slave-hc1k krb5kdc[26593](info): commencing operation
krb5kdc: starting...
W20260926 13:47:21.791606 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:47:21 dist-test-slave-hc1k krb5kdc[26593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:47:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:47:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:47:21.904753 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:36977
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:35461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:36977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:47:22.011238 26606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:22.011480 26606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:22.011521 26606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:22.015709 26606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:47:22.015760 26606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:22.015779 26606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:22.015793 26606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:47:22.015810 26606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:47:22.020886 26606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:35461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:36977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:36977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:22.021840 26606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:22.022786 26606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:47:22.030874 26614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:22.030874 26611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:22.030907 26612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:22.031153 26606 server_base.cc:1034] running on GCE node
I20260926 13:47:22.031705 26606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:22.032729 26606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:22.033896 26606 hybrid_clock.cc:648] HybridClock initialized: now 1790430442033880 us; error 30 us; skew 500 ppm
Sep 26 13:47:22 dist-test-slave-hc1k krb5kdc[26593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430442, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:22.036440 26606 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:47:22.037238 26606 webserver.cc:460] Webserver started at http://127.12.170.126:39763/ using document root <none> and password file <none>
I20260926 13:47:22.037721 26606 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:22.037774 26606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:47:22.037923 26606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:47:22.039830 26606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "eb63e769ec794631860396b694249aaa"
format_stamp: "Formatted at 2026-09-26 13:47:22 on dist-test-slave-hc1k"
server_key: "5d18b47b730b05ac1ccfcc7182f49ad0"
server_key_iv: "fd5b5aeca46f0668e92e60274b99fe06"
server_key_version: "encryptionkey@0"
I20260926 13:47:22.040318 26606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "eb63e769ec794631860396b694249aaa"
format_stamp: "Formatted at 2026-09-26 13:47:22 on dist-test-slave-hc1k"
server_key: "5d18b47b730b05ac1ccfcc7182f49ad0"
server_key_iv: "fd5b5aeca46f0668e92e60274b99fe06"
server_key_version: "encryptionkey@0"
I20260926 13:47:22.043727 26606 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 13:47:22.045892 26621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:22.047034 26606 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:47:22.047127 26606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "eb63e769ec794631860396b694249aaa"
format_stamp: "Formatted at 2026-09-26 13:47:22 on dist-test-slave-hc1k"
server_key: "5d18b47b730b05ac1ccfcc7182f49ad0"
server_key_iv: "fd5b5aeca46f0668e92e60274b99fe06"
server_key_version: "encryptionkey@0"
I20260926 13:47:22.047205 26606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:22.060004 26606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:22.062886 26606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:22.063046 26606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:22.069304 26606 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:36977
I20260926 13:47:22.069370 26673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:36977 every 8 connection(s)
I20260926 13:47:22.070252 26606 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:47:22.072990 26674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:47:22.078298 26674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb63e769ec794631860396b694249aaa: Bootstrap starting.
I20260926 13:47:22.080181 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 26606
I20260926 13:47:22.080287 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:47:22.080561 12969 external_mini_cluster.cc:1428] Setting key 77329e5159212f8636e5e65ba8deb0fa
I20260926 13:47:22.080634 26674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb63e769ec794631860396b694249aaa: Neither blocks nor log segments found. Creating new log.
I20260926 13:47:22.081449 26674 log.cc:824] T 00000000000000000000000000000000 P eb63e769ec794631860396b694249aaa: Log is configured to *not* fsync() on all Append() calls
I20260926 13:47:22.083189 26674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb63e769ec794631860396b694249aaa: No bootstrap required, opened a new log
I20260926 13:47:22.087018 26674 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb63e769ec794631860396b694249aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb63e769ec794631860396b694249aaa" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36977 } }
I20260926 13:47:22.087276 26674 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb63e769ec794631860396b694249aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:47:22.087332 26674 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb63e769ec794631860396b694249aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb63e769ec794631860396b694249aaa, State: Initialized, Role: FOLLOWER
I20260926 13:47:22.087814 26674 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb63e769ec794631860396b694249aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb63e769ec794631860396b694249aaa" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36977 } }
I20260926 13:47:22.087975 26674 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb63e769ec794631860396b694249aaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:47:22.088042 26674 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb63e769ec794631860396b694249aaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:47:22.088182 26674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb63e769ec794631860396b694249aaa [term 0 FOLLOWER]: Advancing to term 1
Sep 26 13:47:22 dist-test-slave-hc1k krb5kdc[26593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:47:22.089026 26674 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb63e769ec794631860396b694249aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb63e769ec794631860396b694249aaa" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36977 } }
I20260926 13:47:22.089375 26674 leader_election.cc:302] T 00000000000000000000000000000000 P eb63e769ec794631860396b694249aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb63e769ec794631860396b694249aaa; no voters:
I20260926 13:47:22.089702 26674 leader_election.cc:288] T 00000000000000000000000000000000 P eb63e769ec794631860396b694249aaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:47:22.089758 26679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb63e769ec794631860396b694249aaa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:47:22.089967 26679 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb63e769ec794631860396b694249aaa [term 1 LEADER]: Becoming Leader. State: Replica: eb63e769ec794631860396b694249aaa, State: Running, Role: LEADER
I20260926 13:47:22.090297 26679 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb63e769ec794631860396b694249aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb63e769ec794631860396b694249aaa" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36977 } }
I20260926 13:47:22.090924 26674 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb63e769ec794631860396b694249aaa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:47:22.091465 26679 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb63e769ec794631860396b694249aaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb63e769ec794631860396b694249aaa. Latest consensus state: current_term: 1 leader_uuid: "eb63e769ec794631860396b694249aaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb63e769ec794631860396b694249aaa" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36977 } } }
I20260926 13:47:22.092219 26679 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb63e769ec794631860396b694249aaa [sys.catalog]: This master's current role is: LEADER
I20260926 13:47:22.092116 26680 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb63e769ec794631860396b694249aaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb63e769ec794631860396b694249aaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb63e769ec794631860396b694249aaa" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36977 } } }
I20260926 13:47:22.092514 26680 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb63e769ec794631860396b694249aaa [sys.catalog]: This master's current role is: LEADER
I20260926 13:47:22.095207 26678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:22.081570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48204 (local address 127.12.170.126:36977)
0926 13:47:22.081996 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:22.082010 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:22.082046 (+ 36us) server_negotiation.cc:407] Connection header received
0926 13:47:22.082671 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:22.082688 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:22.082993 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:22.083286 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:22.084151 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:22.085114 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:22.086284 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:22.086740 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:22.086850 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:22.089688 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:22.089693 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:22.089705 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:22.089735 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:22.091868 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:22.092437 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:22.092443 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:22.092449 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:22.092593 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:22.093144 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:22.093147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:22.093149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:22.093619 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:22.093787 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:22.094045 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 13:47:22.094317 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"spinlock_wait_cycles":11008,"thread_start_us":127,"threads_started":1}
W20260926 13:47:22.095903 26694 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb63e769ec794631860396b694249aaa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:47:22.095963 26694 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:47:22.096041 26689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:47:22.096679 26689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:47:22.098170 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:47:22.101648 26689 catalog_manager.cc:1347] Generated new cluster ID: 4622f6b0eec9494a921f9fb57bcf8326
I20260926 13:47:22.101706 26689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:47:22.105885 26689 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:47:22.107025 26689 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:47:22.118052 26689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb63e769ec794631860396b694249aaa: Generated new TSK 0
I20260926 13:47:22.118772 26689 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:47:22.153620 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 26606
2026-09-26T13:47:22Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1304 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:47:22 dist-test-slave-hc1k krb5kdc[26701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:47:22 dist-test-slave-hc1k krb5kdc[26701](info): set up 2 sockets
Sep 26 13:47:22 dist-test-slave-hc1k krb5kdc[26701](info): commencing operation
krb5kdc: starting...
W20260926 13:47:23.103710 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:47:23 dist-test-slave-hc1k krb5kdc[26701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:47:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:47:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:47:23.218152 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:35935
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:35935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:47:23.326326 26714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:23.326611 26714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:23.326671 26714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:23.330724 26714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:47:23.330789 26714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:23.330816 26714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:23.330835 26714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:47:23.330855 26714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:47:23.336156 26714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:35935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:35935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:23.337182 26714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:23.338184 26714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:47:23.346506 26722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:23.346506 26720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:23.346544 26714 server_base.cc:1034] running on GCE node
W20260926 13:47:23.346506 26719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:23.347318 26714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:23.348445 26714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:23.349601 26714 hybrid_clock.cc:648] HybridClock initialized: now 1790430443349586 us; error 27 us; skew 500 ppm
Sep 26 13:47:23 dist-test-slave-hc1k krb5kdc[26701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430443, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:23.359645 26714 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:47:23.360649 26714 webserver.cc:460] Webserver started at http://127.12.170.126:41015/ using document root <none> and password file <none>
I20260926 13:47:23.361236 26714 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:23.361306 26714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:47:23.361498 26714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:47:23.363514 26714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "b8ba174fc11a4dedafafffe0a841a7bf"
format_stamp: "Formatted at 2026-09-26 13:47:23 on dist-test-slave-hc1k"
server_key: "bd99a3848790e23f967ebec63b9432da"
server_key_iv: "ac49c734ef4f7d58693d8ef477b21fcb"
server_key_version: "encryptionkey@0"
I20260926 13:47:23.364086 26714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "b8ba174fc11a4dedafafffe0a841a7bf"
format_stamp: "Formatted at 2026-09-26 13:47:23 on dist-test-slave-hc1k"
server_key: "bd99a3848790e23f967ebec63b9432da"
server_key_iv: "ac49c734ef4f7d58693d8ef477b21fcb"
server_key_version: "encryptionkey@0"
I20260926 13:47:23.367832 26714 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 13:47:23.370291 26729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:23.371590 26714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:47:23.371784 26714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "b8ba174fc11a4dedafafffe0a841a7bf"
format_stamp: "Formatted at 2026-09-26 13:47:23 on dist-test-slave-hc1k"
server_key: "bd99a3848790e23f967ebec63b9432da"
server_key_iv: "ac49c734ef4f7d58693d8ef477b21fcb"
server_key_version: "encryptionkey@0"
I20260926 13:47:23.371901 26714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:23.400108 26714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:23.403033 26714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:23.403216 26714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:23.409964 26714 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:35935
I20260926 13:47:23.410251 26781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:35935 every 8 connection(s)
I20260926 13:47:23.410954 26714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:47:23.413719 26782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:47:23.414714 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 26714
I20260926 13:47:23.414845 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:47:23.415128 12969 external_mini_cluster.cc:1428] Setting key 97b389aeadbac815bc5494ec11be18f0
I20260926 13:47:23.419629 26782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf: Bootstrap starting.
I20260926 13:47:23.421792 26782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf: Neither blocks nor log segments found. Creating new log.
I20260926 13:47:23.422585 26782 log.cc:824] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf: Log is configured to *not* fsync() on all Append() calls
Sep 26 13:47:23 dist-test-slave-hc1k krb5kdc[26701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:47:23.424695 26782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf: No bootstrap required, opened a new log
I20260926 13:47:23.428561 26782 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ba174fc11a4dedafafffe0a841a7bf" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 35935 } }
I20260926 13:47:23.428778 26782 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:47:23.428810 26782 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ba174fc11a4dedafafffe0a841a7bf, State: Initialized, Role: FOLLOWER
I20260926 13:47:23.428903 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:23.416523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47934 (local address 127.12.170.126:35935)
0926 13:47:23.416997 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:23.417016 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:23.417055 (+ 39us) server_negotiation.cc:407] Connection header received
0926 13:47:23.417693 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:23.417709 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:23.418055 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:23.418371 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:23.419221 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:23.420299 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:23.421168 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:23.421638 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:23.421793 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:23.424467 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:23.424472 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:23.424484 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:23.424519 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:23.426122 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:23.426623 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:23.426627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:23.426632 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:23.426740 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:23.427033 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:23.427035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:23.427036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:23.427406 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:23.427570 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:23.427849 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 13:47:23.428125 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":131,"threads_started":1}
I20260926 13:47:23.429297 26782 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ba174fc11a4dedafafffe0a841a7bf" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 35935 } }
I20260926 13:47:23.429432 26782 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:47:23.429494 26782 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:47:23.429596 26782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:47:23.430680 26782 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ba174fc11a4dedafafffe0a841a7bf" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 35935 } }
I20260926 13:47:23.431034 26782 leader_election.cc:302] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8ba174fc11a4dedafafffe0a841a7bf; no voters:
I20260926 13:47:23.431368 26782 leader_election.cc:288] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:47:23.431447 26787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:47:23.431699 26787 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [term 1 LEADER]: Becoming Leader. State: Replica: b8ba174fc11a4dedafafffe0a841a7bf, State: Running, Role: LEADER
I20260926 13:47:23.432004 26787 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ba174fc11a4dedafafffe0a841a7bf" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 35935 } }
I20260926 13:47:23.432655 26782 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:47:23.433648 26789 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8ba174fc11a4dedafafffe0a841a7bf. Latest consensus state: current_term: 1 leader_uuid: "b8ba174fc11a4dedafafffe0a841a7bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ba174fc11a4dedafafffe0a841a7bf" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 35935 } } }
I20260926 13:47:23.433677 26788 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8ba174fc11a4dedafafffe0a841a7bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ba174fc11a4dedafafffe0a841a7bf" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 35935 } } }
I20260926 13:47:23.433771 26789 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [sys.catalog]: This master's current role is: LEADER
I20260926 13:47:23.433775 26788 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [sys.catalog]: This master's current role is: LEADER
I20260926 13:47:23.434139 26796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:47:23.437259 26796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:47:23.442764 26796 catalog_manager.cc:1347] Generated new cluster ID: 92de00588af14ed29be18c09f59da2d0
I20260926 13:47:23.442847 26796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:47:23.460927 26796 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:47:23.461990 26796 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:47:23.467480 26796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf: Generated new TSK 0
I20260926 13:47:23.468148 26796 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.170.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
I20260926 13:47:23.498384 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:0
--local_ip_for_outbound_sockets=127.12.170.65
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:35935
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:47:23.606428 26808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:23.606685 26808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:23.606727 26808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:23.610752 26808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:23.610819 26808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:23.610877 26808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:47:23.616142 26808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:35935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:23.617161 26808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:23.618096 26808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:47:23.626984 26814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:23.627104 26808 server_base.cc:1034] running on GCE node
W20260926 13:47:23.626997 26813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:23.626988 26816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:23.627557 26808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:23.628196 26808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:23.629355 26808 hybrid_clock.cc:648] HybridClock initialized: now 1790430443629337 us; error 29 us; skew 500 ppm
Sep 26 13:47:23 dist-test-slave-hc1k krb5kdc[26701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430443, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:23.631901 26808 init.cc:377] Logged in from keytab as kudu/127.12.170.65@KRBTEST.COM (short username kudu)
I20260926 13:47:23.632772 26808 webserver.cc:460] Webserver started at http://127.12.170.65:35041/ using document root <none> and password file <none>
I20260926 13:47:23.633358 26808 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:23.633424 26808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:47:23.633622 26808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:47:23.635669 26808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/data/instance:
uuid: "238606c7eb7c42fdb459e36d055dd84b"
format_stamp: "Formatted at 2026-09-26 13:47:23 on dist-test-slave-hc1k"
server_key: "dd88941eacd99e197e057daa65c20892"
server_key_iv: "8919bcd94b13eaeabce221940b6f60bb"
server_key_version: "encryptionkey@0"
I20260926 13:47:23.636225 26808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance:
uuid: "238606c7eb7c42fdb459e36d055dd84b"
format_stamp: "Formatted at 2026-09-26 13:47:23 on dist-test-slave-hc1k"
server_key: "dd88941eacd99e197e057daa65c20892"
server_key_iv: "8919bcd94b13eaeabce221940b6f60bb"
server_key_version: "encryptionkey@0"
I20260926 13:47:23.639779 26808 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:47:23.642151 26823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:23.643579 26808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:47:23.643715 26808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "238606c7eb7c42fdb459e36d055dd84b"
format_stamp: "Formatted at 2026-09-26 13:47:23 on dist-test-slave-hc1k"
server_key: "dd88941eacd99e197e057daa65c20892"
server_key_iv: "8919bcd94b13eaeabce221940b6f60bb"
server_key_version: "encryptionkey@0"
I20260926 13:47:23.643818 26808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:23.666882 26808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:23.669844 26808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:23.670035 26808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:23.670619 26808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:47:23.671433 26808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:47:23.671478 26808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:23.671530 26808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:47:23.671559 26808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:23.679783 26808 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:39299
I20260926 13:47:23.679802 26936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:39299 every 8 connection(s)
I20260926 13:47:23.680610 26808 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:47:23.684791 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 26808
I20260926 13:47:23.684888 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance
I20260926 13:47:23.685173 12969 external_mini_cluster.cc:1428] Setting key f7a2be3486f3b433542f57804fe822b8
Sep 26 13:47:23 dist-test-slave-hc1k krb5kdc[26701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430443, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.170.66@KRBTEST.COM; defaulting to no policy
I20260926 13:47:23.694013 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:23.682281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:53725 (local address 127.12.170.126:35935)
0926 13:47:23.682401 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:23.682405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:23.683226 (+ 821us) server_negotiation.cc:407] Connection header received
0926 13:47:23.684582 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:23.684585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:23.684640 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:23.684727 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:23.686198 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:23.686728 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:23.687935 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:23.688181 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:23.688260 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:23.690783 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:23.690787 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:23.690789 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:23.690815 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:23.692147 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:23.692597 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:23.692601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:23.692603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:23.692679 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:23.692971 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:23.692974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:23.692976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:23.693155 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:23.693239 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:23.693715 (+ 476us) server_negotiation.cc:299] Negotiation successful
0926 13:47:23.693852 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:47:23.694828 26939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:23.682643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.65:53725)
0926 13:47:23.683088 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:23.683130 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:23.684378 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:23.684926 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:23.684936 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:23.685419 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:23.686061 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:23.686077 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:23.686846 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:23.686849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:23.687807 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:23.687819 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:23.688318 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:23.688322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:23.688510 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:23.689084 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:23.689102 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:23.690626 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:23.692248 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:23.692253 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:23.692268 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:23.692509 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:23.692756 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:23.692759 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:23.692761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:23.692889 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:23.693251 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:23.693257 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:23.693497 (+ 240us) client_negotiation.cc:769] Sending connection context
0926 13:47:23.693663 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 13:47:23.693986 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":106,"threads_started":1}
I20260926 13:47:23.696132 26937 heartbeater.cc:347] Connected to a master server at 127.12.170.126:35935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.66@KRBTEST.COM" created.
I20260926 13:47:23.696376 26937 heartbeater.cc:464] Registering TS with master...
I20260926 13:47:23.696854 26937 heartbeater.cc:511] Master 127.12.170.126:35935 requested a full tablet report, sending...
I20260926 13:47:23.698488 26746 ts_manager.cc:194] Registered new tserver with Master: 238606c7eb7c42fdb459e36d055dd84b (127.12.170.65:39299)
I20260926 13:47:23.699715 26746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.65@KRBTEST.COM'} at 127.12.170.65:53725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
I20260926 13:47:23.710949 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:0
--local_ip_for_outbound_sockets=127.12.170.66
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:35935
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:47:23.817797 26942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:23.818048 26942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:23.818089 26942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:23.822115 26942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:23.822172 26942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:23.822234 26942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:47:23.827426 26942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:35935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:23.828418 26942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:23.829384 26942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:47:23.838766 26947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:23.838804 26942 server_base.cc:1034] running on GCE node
W20260926 13:47:23.838769 26950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:23.838817 26948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:23.839411 26942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:23.840056 26942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:23.841217 26942 hybrid_clock.cc:648] HybridClock initialized: now 1790430443841199 us; error 30 us; skew 500 ppm
Sep 26 13:47:23 dist-test-slave-hc1k krb5kdc[26701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430443, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:23.843823 26942 init.cc:377] Logged in from keytab as kudu/127.12.170.66@KRBTEST.COM (short username kudu)
I20260926 13:47:23.844667 26942 webserver.cc:460] Webserver started at http://127.12.170.66:33833/ using document root <none> and password file <none>
I20260926 13:47:23.845180 26942 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:23.845234 26942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:47:23.845394 26942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:47:23.847364 26942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/data/instance:
uuid: "3aee7b81806b4d9ebaff29a885d69ca9"
format_stamp: "Formatted at 2026-09-26 13:47:23 on dist-test-slave-hc1k"
server_key: "c63550c4996d4158616a004915edb31e"
server_key_iv: "6c02efe2825ecadcaf72ceb821b45515"
server_key_version: "encryptionkey@0"
I20260926 13:47:23.847903 26942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance:
uuid: "3aee7b81806b4d9ebaff29a885d69ca9"
format_stamp: "Formatted at 2026-09-26 13:47:23 on dist-test-slave-hc1k"
server_key: "c63550c4996d4158616a004915edb31e"
server_key_iv: "6c02efe2825ecadcaf72ceb821b45515"
server_key_version: "encryptionkey@0"
I20260926 13:47:23.851361 26942 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260926 13:47:23.853667 26957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:23.854909 26942 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:47:23.855013 26942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "3aee7b81806b4d9ebaff29a885d69ca9"
format_stamp: "Formatted at 2026-09-26 13:47:23 on dist-test-slave-hc1k"
server_key: "c63550c4996d4158616a004915edb31e"
server_key_iv: "6c02efe2825ecadcaf72ceb821b45515"
server_key_version: "encryptionkey@0"
I20260926 13:47:23.855127 26942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:23.867703 26942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:23.870447 26942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:23.870622 26942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:23.871209 26942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:47:23.872036 26942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:47:23.872100 26942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:23.872159 26942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:47:23.872180 26942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:23.880355 26942 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:40559
I20260926 13:47:23.880393 27070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:40559 every 8 connection(s)
I20260926 13:47:23.881219 26942 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:47:23.886722 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 26942
I20260926 13:47:23.886816 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance
I20260926 13:47:23.887058 12969 external_mini_cluster.cc:1428] Setting key ec1f7aeeb3476b724b402a633fc79934
Sep 26 13:47:23 dist-test-slave-hc1k krb5kdc[26701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430443, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:47:23.894865 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:23.882892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:59763 (local address 127.12.170.126:35935)
0926 13:47:23.883028 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:23.883033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:23.883901 (+ 868us) server_negotiation.cc:407] Connection header received
0926 13:47:23.885207 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:23.885210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:23.885258 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:23.885329 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:23.886452 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:23.887020 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:23.888428 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:23.888722 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:23.888824 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:23.891166 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:23.891170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:23.891172 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:23.891200 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:23.892721 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:23.893227 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:23.893231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:23.893233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:23.893322 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:23.893700 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:23.893703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:23.893705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:23.893940 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:23.894041 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:23.894574 (+ 533us) server_negotiation.cc:299] Negotiation successful
0926 13:47:23.894695 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 13:47:23.895632 27073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:23.883300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.66:59763)
0926 13:47:23.883761 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:23.883803 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:23.885004 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:23.885477 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:23.885483 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:23.885814 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:23.886311 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:23.886324 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:23.887147 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:23.887151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:23.888321 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:23.888329 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:23.888861 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:23.888865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:23.889021 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:23.889567 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:23.889582 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:23.891005 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:23.892839 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:23.892843 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:23.892856 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:23.893134 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:23.893442 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:23.893445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:23.893446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:23.893605 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:23.894076 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:23.894081 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:23.894332 (+ 251us) client_negotiation.cc:769] Sending connection context
0926 13:47:23.894492 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 13:47:23.894755 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.12.170.67@KRBTEST.COM; defaulting to no policy
I20260926 13:47:23.896883 27071 heartbeater.cc:347] Connected to a master server at 127.12.170.126:35935
I20260926 13:47:23.897150 27071 heartbeater.cc:464] Registering TS with master...
I20260926 13:47:23.897619 27071 heartbeater.cc:511] Master 127.12.170.126:35935 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.67@KRBTEST.COM" created.
I20260926 13:47:23.898751 26746 ts_manager.cc:194] Registered new tserver with Master: 3aee7b81806b4d9ebaff29a885d69ca9 (127.12.170.66:40559)
I20260926 13:47:23.899459 26746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.66@KRBTEST.COM'} at 127.12.170.66:59763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
I20260926 13:47:23.914119 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:0
--local_ip_for_outbound_sockets=127.12.170.67
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:35935
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:47:24.020437 27076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:24.020679 27076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:24.020736 27076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:24.024649 27076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:24.024698 27076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:24.024796 27076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:47:24.029855 27076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:35935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:24.030798 27076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:24.031780 27076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:47:24.040802 27082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:24.040823 27081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:24.040885 27076 server_base.cc:1034] running on GCE node
W20260926 13:47:24.040834 27084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:24.041371 27076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:24.041941 27076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:24.043095 27076 hybrid_clock.cc:648] HybridClock initialized: now 1790430444043083 us; error 41 us; skew 500 ppm
Sep 26 13:47:24 dist-test-slave-hc1k krb5kdc[26701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430444, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:24.045652 27076 init.cc:377] Logged in from keytab as kudu/127.12.170.67@KRBTEST.COM (short username kudu)
I20260926 13:47:24.046420 27076 webserver.cc:460] Webserver started at http://127.12.170.67:41331/ using document root <none> and password file <none>
I20260926 13:47:24.046908 27076 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:24.046979 27076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:47:24.047154 27076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:47:24.049126 27076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/data/instance:
uuid: "7466e77c905546d8a1be7fadd7a5817b"
format_stamp: "Formatted at 2026-09-26 13:47:24 on dist-test-slave-hc1k"
server_key: "7545e64d9744de102e6566d9b29260b8"
server_key_iv: "c5abd7297a1f759f7d65dc25df72f6e3"
server_key_version: "encryptionkey@0"
I20260926 13:47:24.049598 27076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance:
uuid: "7466e77c905546d8a1be7fadd7a5817b"
format_stamp: "Formatted at 2026-09-26 13:47:24 on dist-test-slave-hc1k"
server_key: "7545e64d9744de102e6566d9b29260b8"
server_key_iv: "c5abd7297a1f759f7d65dc25df72f6e3"
server_key_version: "encryptionkey@0"
I20260926 13:47:24.052976 27076 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:47:24.055199 27091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:24.056420 27076 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:47:24.056510 27076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "7466e77c905546d8a1be7fadd7a5817b"
format_stamp: "Formatted at 2026-09-26 13:47:24 on dist-test-slave-hc1k"
server_key: "7545e64d9744de102e6566d9b29260b8"
server_key_iv: "c5abd7297a1f759f7d65dc25df72f6e3"
server_key_version: "encryptionkey@0"
I20260926 13:47:24.056610 27076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:24.074527 27076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:24.077751 27076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:24.077922 27076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:24.078495 27076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:47:24.079290 27076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:47:24.079334 27076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:24.079377 27076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:47:24.079407 27076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:24.087321 27076 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:36135
I20260926 13:47:24.087373 27204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:36135 every 8 connection(s)
I20260926 13:47:24.088171 27076 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:47:24.089813 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 27076
I20260926 13:47:24.089906 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance
I20260926 13:47:24.090188 12969 external_mini_cluster.cc:1428] Setting key 5f6fcc67bd6ef43a044f4cf398b84a92
Sep 26 13:47:24 dist-test-slave-hc1k krb5kdc[26701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430444, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:47:24.100553 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:24.089900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:38731 (local address 127.12.170.126:35935)
0926 13:47:24.090037 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:24.090042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:24.090897 (+ 855us) server_negotiation.cc:407] Connection header received
0926 13:47:24.092267 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:24.092269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:24.092313 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:24.092375 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:24.093400 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.093911 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.094998 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.095219 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.095297 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:24.097457 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:24.097460 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:24.097462 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:24.097485 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:24.098755 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:24.099180 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:24.099183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:24.099184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:24.099267 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:24.099557 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:24.099560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:24.099561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:24.099791 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:24.099875 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:24.100323 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 13:47:24.100429 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 13:47:24.101312 27207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:24.090278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.67:38731)
0926 13:47:24.090754 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:24.090805 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:24.092084 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:24.092501 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:24.092507 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:24.092835 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:24.093299 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:24.093312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.093992 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.093994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:24.094895 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:24.094904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.095330 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.095332 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:24.095468 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:24.095967 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:24.095983 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:24.097314 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:24.098840 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:24.098844 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:24.098854 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:24.099076 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:24.099342 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:24.099344 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:24.099346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:24.099458 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:24.099871 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:24.099876 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:24.100091 (+ 215us) client_negotiation.cc:769] Sending connection context
0926 13:47:24.100278 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 13:47:24.100511 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":118,"threads_started":1}
I20260926 13:47:24.102451 27205 heartbeater.cc:347] Connected to a master server at 127.12.170.126:35935
I20260926 13:47:24.102692 27205 heartbeater.cc:464] Registering TS with master...
I20260926 13:47:24.103206 27205 heartbeater.cc:511] Master 127.12.170.126:35935 requested a full tablet report, sending...
I20260926 13:47:24.104197 26746 ts_manager.cc:194] Registered new tserver with Master: 7466e77c905546d8a1be7fadd7a5817b (127.12.170.67:36135)
I20260926 13:47:24.104746 26746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.67@KRBTEST.COM'} at 127.12.170.67:38731
I20260926 13:47:24.115219 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:47:24.123693 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:24.117071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47944 (local address 127.12.170.126:35935)
0926 13:47:24.117211 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:24.117216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:24.117352 (+ 136us) server_negotiation.cc:407] Connection header received
0926 13:47:24.117469 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:24.117473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:24.117514 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:24.117574 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:24.118255 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.118756 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.119547 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.119799 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.119883 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:24.121090 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:24.121094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:24.121097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:24.121123 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:24.122192 (+ 1069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:24.122561 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:24.122564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:24.122566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:24.122638 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:24.122865 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:24.122869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:24.122870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:24.123079 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:24.123203 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:24.123410 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 13:47:24.123516 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:47:24.126291 26746 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:47:24.129256 26746 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:47:24.142961 27216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:24.137307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:40559 (local address 127.0.0.1:51114)
0926 13:47:24.137756 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:24.137802 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:24.137919 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:24.138827 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:24.138833 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:24.138855 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:47:24.138985 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:24.138991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.140256 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.140259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:24.141419 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:24.141429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.142466 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.142470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:24.142569 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:24.142610 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 13:47:24.142705 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 13:47:24.142819 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":83,"threads_started":1}
I20260926 13:47:24.143247 27218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:24.137530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:36135 (local address 127.0.0.1:55832)
0926 13:47:24.137989 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:24.138002 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:24.138086 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:24.139129 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:24.139132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:24.139150 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:47:24.139249 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:24.139255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.140706 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.140709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:24.141919 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:24.141928 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.142897 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.142900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:24.142977 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:24.142997 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 13:47:24.143066 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 13:47:24.143131 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":38,"thread_start_us":95,"threads_started":1}
I20260926 13:47:24.143303 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:24.138087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55832 (local address 127.12.170.67:36135)
0926 13:47:24.138699 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:24.138709 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:24.138731 (+ 22us) server_negotiation.cc:407] Connection header received
0926 13:47:24.138801 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:24.138806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:24.138983 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:24.139131 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:47:24.139375 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.140559 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.142042 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.142803 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.142897 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:24.143017 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:24.143092 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 13:47:24.143178 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":122,"threads_started":1}
I20260926 13:47:24.143421 27221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:24.138417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59308 (local address 127.12.170.65:39299)
0926 13:47:24.138827 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:24.138838 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:24.138861 (+ 23us) server_negotiation.cc:407] Connection header received
0926 13:47:24.138932 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:24.138937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:24.139085 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:24.139225 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:47:24.139505 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.140613 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.142000 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.142565 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.142672 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:24.142803 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:24.143202 (+ 399us) server_negotiation.cc:299] Negotiation successful
0926 13:47:24.143292 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":129,"threads_started":1}
I20260926 13:47:24.143121 27219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:24.137982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:39299 (local address 127.0.0.1:59308)
0926 13:47:24.138222 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:24.138235 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:24.138311 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:24.139292 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:24.139294 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:24.139311 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:47:24.139399 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:24.139404 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.140740 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.140744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:24.141895 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:24.141904 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.142687 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.142691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:24.142784 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:24.142807 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 13:47:24.142874 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 13:47:24.142938 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":61,"thread_start_us":116,"threads_started":1}
I20260926 13:47:24.143877 27217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:24.137456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51114 (local address 127.12.170.66:40559)
0926 13:47:24.138313 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:24.138325 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:24.138348 (+ 23us) server_negotiation.cc:407] Connection header received
0926 13:47:24.138425 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:24.138431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:24.138658 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:24.139132 (+ 474us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:47:24.139280 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.140141 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.141549 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.142354 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.142452 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:24.142590 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:24.143620 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0926 13:47:24.143745 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":170,"threads_started":1}
I20260926 13:47:24.146597 27005 tablet_service.cc:1467] Processing CreateTablet for tablet 5414033abcf94fdea69f21ba91630a43 (DEFAULT_TABLE table=test-table [id=2bc2c0edc96148da88eb6176b8c08436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:47:24.146652 26871 tablet_service.cc:1467] Processing CreateTablet for tablet 5414033abcf94fdea69f21ba91630a43 (DEFAULT_TABLE table=test-table [id=2bc2c0edc96148da88eb6176b8c08436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:47:24.146652 27139 tablet_service.cc:1467] Processing CreateTablet for tablet 5414033abcf94fdea69f21ba91630a43 (DEFAULT_TABLE table=test-table [id=2bc2c0edc96148da88eb6176b8c08436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:47:24.147532 27005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5414033abcf94fdea69f21ba91630a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:47:24.147610 26871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5414033abcf94fdea69f21ba91630a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:47:24.147619 27139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5414033abcf94fdea69f21ba91630a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:47:24.153890 27222 tablet_bootstrap.cc:492] T 5414033abcf94fdea69f21ba91630a43 P 3aee7b81806b4d9ebaff29a885d69ca9: Bootstrap starting.
I20260926 13:47:24.154683 27223 tablet_bootstrap.cc:492] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b: Bootstrap starting.
I20260926 13:47:24.155998 27224 tablet_bootstrap.cc:492] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b: Bootstrap starting.
I20260926 13:47:24.156435 27222 tablet_bootstrap.cc:654] T 5414033abcf94fdea69f21ba91630a43 P 3aee7b81806b4d9ebaff29a885d69ca9: Neither blocks nor log segments found. Creating new log.
I20260926 13:47:24.156473 27223 tablet_bootstrap.cc:654] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b: Neither blocks nor log segments found. Creating new log.
I20260926 13:47:24.157306 27222 log.cc:824] T 5414033abcf94fdea69f21ba91630a43 P 3aee7b81806b4d9ebaff29a885d69ca9: Log is configured to *not* fsync() on all Append() calls
I20260926 13:47:24.157418 27223 log.cc:824] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b: Log is configured to *not* fsync() on all Append() calls
I20260926 13:47:24.158677 27224 tablet_bootstrap.cc:654] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b: Neither blocks nor log segments found. Creating new log.
I20260926 13:47:24.159408 27222 tablet_bootstrap.cc:492] T 5414033abcf94fdea69f21ba91630a43 P 3aee7b81806b4d9ebaff29a885d69ca9: No bootstrap required, opened a new log
I20260926 13:47:24.159408 27223 tablet_bootstrap.cc:492] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b: No bootstrap required, opened a new log
I20260926 13:47:24.159396 27224 log.cc:824] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b: Log is configured to *not* fsync() on all Append() calls
I20260926 13:47:24.159595 27222 ts_tablet_manager.cc:1397] T 5414033abcf94fdea69f21ba91630a43 P 3aee7b81806b4d9ebaff29a885d69ca9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 13:47:24.159595 27223 ts_tablet_manager.cc:1397] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 13:47:24.161278 27224 tablet_bootstrap.cc:492] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b: No bootstrap required, opened a new log
I20260926 13:47:24.161443 27224 ts_tablet_manager.cc:1397] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260926 13:47:24.163357 27223 raft_consensus.cc:348] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } } peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } }
I20260926 13:47:24.163357 27222 raft_consensus.cc:348] T 5414033abcf94fdea69f21ba91630a43 P 3aee7b81806b4d9ebaff29a885d69ca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } } peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } }
I20260926 13:47:24.163618 27222 raft_consensus.cc:374] T 5414033abcf94fdea69f21ba91630a43 P 3aee7b81806b4d9ebaff29a885d69ca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:47:24.163618 27223 raft_consensus.cc:374] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:47:24.163693 27222 raft_consensus.cc:729] T 5414033abcf94fdea69f21ba91630a43 P 3aee7b81806b4d9ebaff29a885d69ca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aee7b81806b4d9ebaff29a885d69ca9, State: Initialized, Role: FOLLOWER
I20260926 13:47:24.163693 27223 raft_consensus.cc:729] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 238606c7eb7c42fdb459e36d055dd84b, State: Initialized, Role: FOLLOWER
I20260926 13:47:24.164139 27223 consensus_queue.cc:260] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } } peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } }
I20260926 13:47:24.164139 27222 consensus_queue.cc:260] T 5414033abcf94fdea69f21ba91630a43 P 3aee7b81806b4d9ebaff29a885d69ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } } peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } }
I20260926 13:47:24.164235 27224 raft_consensus.cc:348] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } } peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } }
I20260926 13:47:24.164470 27224 raft_consensus.cc:374] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:47:24.164525 27224 raft_consensus.cc:729] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7466e77c905546d8a1be7fadd7a5817b, State: Initialized, Role: FOLLOWER
I20260926 13:47:24.165130 27222 ts_tablet_manager.cc:1428] T 5414033abcf94fdea69f21ba91630a43 P 3aee7b81806b4d9ebaff29a885d69ca9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 13:47:24.165148 27223 ts_tablet_manager.cc:1428] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 13:47:24.165154 27071 heartbeater.cc:503] Master 127.12.170.126:35935 was elected leader, sending a full tablet report...
I20260926 13:47:24.164947 27224 consensus_queue.cc:260] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } } peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } }
I20260926 13:47:24.165437 26937 heartbeater.cc:503] Master 127.12.170.126:35935 was elected leader, sending a full tablet report...
I20260926 13:47:24.166189 27224 ts_tablet_manager.cc:1428] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260926 13:47:24.166471 27205 heartbeater.cc:503] Master 127.12.170.126:35935 was elected leader, sending a full tablet report...
W20260926 13:47:24.181663 26938 tablet.cc:2369] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:47:24.339084 27206 tablet.cc:2369] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:47:24.364667 27230 raft_consensus.cc:482] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:47:24.364877 27230 raft_consensus.cc:504] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } } peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } }
I20260926 13:47:24.365972 27230 leader_election.cc:288] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aee7b81806b4d9ebaff29a885d69ca9 (127.12.170.66:40559), 238606c7eb7c42fdb459e36d055dd84b (127.12.170.65:39299)
I20260926 13:47:24.370275 27217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:24.366355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:40087 (local address 127.12.170.66:40559)
0926 13:47:24.366476 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:24.366481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:24.366496 (+ 15us) server_negotiation.cc:407] Connection header received
0926 13:47:24.366618 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:24.366622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:24.366666 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:24.366726 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:47:24.367132 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.367925 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.369327 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.369714 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.369818 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:24.369873 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:24.370083 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 13:47:24.370155 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:47:24.370275 27221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:24.366668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:48335 (local address 127.12.170.65:39299)
0926 13:47:24.366814 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:24.366818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:24.367003 (+ 185us) server_negotiation.cc:407] Connection header received
0926 13:47:24.367077 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:24.367080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:24.367132 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:24.367220 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:47:24.367438 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.368129 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.369358 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.369734 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.369831 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:24.369879 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:24.370080 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 13:47:24.370149 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 13:47:24.370275 27207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:24.366262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:40559 (local address 127.12.170.67:40087)
0926 13:47:24.366384 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:24.366400 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:24.366511 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:24.366851 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:24.366854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:24.366876 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:47:24.367010 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:24.367016 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.368075 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.368078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:24.369217 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:24.369226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.369821 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.369824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:24.369926 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:24.369990 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 13:47:24.370057 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 13:47:24.370142 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":111}
I20260926 13:47:24.370287 27231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:24.366542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:39299 (local address 127.12.170.67:48335)
0926 13:47:24.366877 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:24.366889 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:24.366982 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:24.367243 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:24.367245 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:24.367260 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:47:24.367348 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:24.367352 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.368230 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.368233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:24.369253 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:24.369263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.369832 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.369835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:24.369936 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:24.369998 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 13:47:24.370058 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 13:47:24.370135 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":52,"thread_start_us":144,"threads_started":1}
I20260926 13:47:24.370895 26891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5414033abcf94fdea69f21ba91630a43" candidate_uuid: "7466e77c905546d8a1be7fadd7a5817b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "238606c7eb7c42fdb459e36d055dd84b" is_pre_election: true
I20260926 13:47:24.370918 27025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5414033abcf94fdea69f21ba91630a43" candidate_uuid: "7466e77c905546d8a1be7fadd7a5817b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" is_pre_election: true
I20260926 13:47:24.371222 26891 raft_consensus.cc:2454] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7466e77c905546d8a1be7fadd7a5817b in term 0.
I20260926 13:47:24.371222 27025 raft_consensus.cc:2454] T 5414033abcf94fdea69f21ba91630a43 P 3aee7b81806b4d9ebaff29a885d69ca9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7466e77c905546d8a1be7fadd7a5817b in term 0.
I20260926 13:47:24.371688 27093 leader_election.cc:302] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 238606c7eb7c42fdb459e36d055dd84b, 7466e77c905546d8a1be7fadd7a5817b; no voters:
I20260926 13:47:24.371927 27230 raft_consensus.cc:2781] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:47:24.372008 27230 raft_consensus.cc:482] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:47:24.372056 27230 raft_consensus.cc:3037] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:47:24.373008 27230 raft_consensus.cc:504] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } } peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } }
I20260926 13:47:24.373309 27230 leader_election.cc:288] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b [CANDIDATE]: Term 1 election: Requested vote from peers 3aee7b81806b4d9ebaff29a885d69ca9 (127.12.170.66:40559), 238606c7eb7c42fdb459e36d055dd84b (127.12.170.65:39299)
I20260926 13:47:24.373641 27025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5414033abcf94fdea69f21ba91630a43" candidate_uuid: "7466e77c905546d8a1be7fadd7a5817b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aee7b81806b4d9ebaff29a885d69ca9"
I20260926 13:47:24.373703 26891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5414033abcf94fdea69f21ba91630a43" candidate_uuid: "7466e77c905546d8a1be7fadd7a5817b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "238606c7eb7c42fdb459e36d055dd84b"
I20260926 13:47:24.373761 27025 raft_consensus.cc:3037] T 5414033abcf94fdea69f21ba91630a43 P 3aee7b81806b4d9ebaff29a885d69ca9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:47:24.373807 26891 raft_consensus.cc:3037] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:47:24.374943 26891 raft_consensus.cc:2454] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7466e77c905546d8a1be7fadd7a5817b in term 1.
I20260926 13:47:24.374943 27025 raft_consensus.cc:2454] T 5414033abcf94fdea69f21ba91630a43 P 3aee7b81806b4d9ebaff29a885d69ca9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7466e77c905546d8a1be7fadd7a5817b in term 1.
I20260926 13:47:24.375203 27093 leader_election.cc:302] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aee7b81806b4d9ebaff29a885d69ca9, 7466e77c905546d8a1be7fadd7a5817b; no voters:
I20260926 13:47:24.375360 27230 raft_consensus.cc:2781] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:47:24.375576 27230 raft_consensus.cc:686] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b [term 1 LEADER]: Becoming Leader. State: Replica: 7466e77c905546d8a1be7fadd7a5817b, State: Running, Role: LEADER
I20260926 13:47:24.375936 27230 consensus_queue.cc:237] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } } peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } }
I20260926 13:47:24.378871 26744 catalog_manager.cc:5495] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7466e77c905546d8a1be7fadd7a5817b (127.12.170.67). New cstate: current_term: 1 leader_uuid: "7466e77c905546d8a1be7fadd7a5817b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } health_report { overall_health: HEALTHY } } }
W20260926 13:47:24.382269 27072 tablet.cc:2369] T 5414033abcf94fdea69f21ba91630a43 P 3aee7b81806b4d9ebaff29a885d69ca9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:47:24.433202 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:24.429953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55838 (local address 127.12.170.67:36135)
0926 13:47:24.430066 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:24.430070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:24.430100 (+ 30us) server_negotiation.cc:407] Connection header received
0926 13:47:24.430174 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:24.430177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:24.430242 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:24.430319 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:47:24.430526 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.431025 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.431809 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.432010 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.432079 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:24.432337 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:47:24.432774 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:47:24.432910 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:24.432991 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 13:47:24.433049 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:47:24.438881 26891 raft_consensus.cc:1261] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b [term 1 FOLLOWER]: Refusing update from remote peer 7466e77c905546d8a1be7fadd7a5817b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:47:24.438881 27025 raft_consensus.cc:1261] T 5414033abcf94fdea69f21ba91630a43 P 3aee7b81806b4d9ebaff29a885d69ca9 [term 1 FOLLOWER]: Refusing update from remote peer 7466e77c905546d8a1be7fadd7a5817b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:47:24.439625 27232 consensus_queue.cc:1035] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:47:24.439852 27230 consensus_queue.cc:1035] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b [LEADER]: Connected to new peer: Peer: permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:47:24.447296 27235 mvcc.cc:205] Tried to move back new op lower bound from 7333603100414746624 to 7333603100166385664. Current Snapshot: MvccSnapshot[applied={T|T < 7333603100414746624}]
I20260926 13:47:24.449574 27237 mvcc.cc:205] Tried to move back new op lower bound from 7333603100414746624 to 7333603100166385664. Current Snapshot: MvccSnapshot[applied={T|T < 7333603100414746624}]
I20260926 13:47:24.454805 26745 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:47:24.454947 26745 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:47:24.457207 26745 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf: Sending DeleteTablet for 3 replicas of tablet 5414033abcf94fdea69f21ba91630a43
I20260926 13:47:24.457893 27005 tablet_service.cc:1514] Processing DeleteTablet for tablet 5414033abcf94fdea69f21ba91630a43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:47:24 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:51114
I20260926 13:47:24.457896 26871 tablet_service.cc:1514] Processing DeleteTablet for tablet 5414033abcf94fdea69f21ba91630a43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:47:24 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:59308
I20260926 13:47:24.458365 27139 tablet_service.cc:1514] Processing DeleteTablet for tablet 5414033abcf94fdea69f21ba91630a43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:47:24 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:55832
I20260926 13:47:24.458447 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 26808
I20260926 13:47:24.459146 27246 tablet_replica.cc:330] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b: stopping tablet replica
I20260926 13:47:24.459419 27246 raft_consensus.cc:2229] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:47:24.459699 27245 tablet_replica.cc:330] T 5414033abcf94fdea69f21ba91630a43 P 3aee7b81806b4d9ebaff29a885d69ca9: stopping tablet replica
I20260926 13:47:24.459780 27246 raft_consensus.cc:2258] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:47:24.459995 27245 raft_consensus.cc:2229] T 5414033abcf94fdea69f21ba91630a43 P 3aee7b81806b4d9ebaff29a885d69ca9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:47:24.460326 27245 raft_consensus.cc:2258] T 5414033abcf94fdea69f21ba91630a43 P 3aee7b81806b4d9ebaff29a885d69ca9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:47:24.461205 27246 ts_tablet_manager.cc:1905] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:47:24.461879 27245 ts_tablet_manager.cc:1905] T 5414033abcf94fdea69f21ba91630a43 P 3aee7b81806b4d9ebaff29a885d69ca9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:47:24.463842 27246 ts_tablet_manager.cc:1918] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:47:24.463949 27246 log.cc:1196] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/5414033abcf94fdea69f21ba91630a43
I20260926 13:47:24.464278 27246 ts_tablet_manager.cc:1939] T 5414033abcf94fdea69f21ba91630a43 P 7466e77c905546d8a1be7fadd7a5817b: Deleting consensus metadata
I20260926 13:47:24.464692 27245 ts_tablet_manager.cc:1918] T 5414033abcf94fdea69f21ba91630a43 P 3aee7b81806b4d9ebaff29a885d69ca9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:47:24.464851 27245 log.cc:1196] T 5414033abcf94fdea69f21ba91630a43 P 3aee7b81806b4d9ebaff29a885d69ca9: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/wal/wals/5414033abcf94fdea69f21ba91630a43
I20260926 13:47:24.465214 27245 ts_tablet_manager.cc:1939] T 5414033abcf94fdea69f21ba91630a43 P 3aee7b81806b4d9ebaff29a885d69ca9: Deleting consensus metadata
W20260926 13:47:24.465555 26731 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.170.65:39299 (error 108)
I20260926 13:47:24.465807 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 26942
I20260926 13:47:24.465871 26733 catalog_manager.cc:4841] TS 7466e77c905546d8a1be7fadd7a5817b (127.12.170.67:36135): tablet 5414033abcf94fdea69f21ba91630a43 (table test-table [id=2bc2c0edc96148da88eb6176b8c08436]) successfully deleted
I20260926 13:47:24.468031 27219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:24.467775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:39299 (local address 127.0.0.1:59324)
0926 13:47:24.467895 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:24.467960 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.65:39299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 13:47:24.468191 26731 catalog_manager.cc:4584] TS 238606c7eb7c42fdb459e36d055dd84b (127.12.170.65:39299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5414033abcf94fdea69f21ba91630a43: Network error: Client connection negotiation failed: client connection to 127.12.170.65:39299: connect: Connection refused (error 111)
I20260926 13:47:24.471978 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 27076
I20260926 13:47:24.473688 27219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:24.472125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:40559 (local address 127.0.0.1:51116)
0926 13:47:24.473566 (+ 1441us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:24.473614 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.66:40559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1369}
I20260926 13:47:24.476882 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 26714
I20260926 13:47:24.481909 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:35935
--webserver_interface=127.12.170.126
--webserver_port=41015
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:35935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:47:24.588415 27248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:24.588683 27248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:24.588739 27248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:24.592692 27248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:47:24.592757 27248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:24.592785 27248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:24.592806 27248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:47:24.592833 27248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:47:24.598038 27248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:35935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:35935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=41015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:24.599058 27248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:24.600077 27248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:47:24.612347 27248 server_base.cc:1034] running on GCE node
W20260926 13:47:24.612262 27255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:24.612303 27257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:24.612265 27254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:24.613180 27248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:24.614274 27248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:24.615429 27248 hybrid_clock.cc:648] HybridClock initialized: now 1790430444615417 us; error 29 us; skew 500 ppm
Sep 26 13:47:24 dist-test-slave-hc1k krb5kdc[26701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430444, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:24.618206 27248 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:47:24.619076 27248 webserver.cc:460] Webserver started at http://127.12.170.126:41015/ using document root <none> and password file <none>
I20260926 13:47:24.619634 27248 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:24.619743 27248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:47:24.623575 27248 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:47:24.625670 27264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:24.626919 27248 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:47:24.627067 27248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "b8ba174fc11a4dedafafffe0a841a7bf"
format_stamp: "Formatted at 2026-09-26 13:47:23 on dist-test-slave-hc1k"
server_key: "bd99a3848790e23f967ebec63b9432da"
server_key_iv: "ac49c734ef4f7d58693d8ef477b21fcb"
server_key_version: "encryptionkey@0"
I20260926 13:47:24.627748 27248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:24.638723 27248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:24.641628 27248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:24.641817 27248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:24.649005 27248 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:35935
I20260926 13:47:24.649253 27316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:35935 every 8 connection(s)
I20260926 13:47:24.650128 27248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:47:24.654385 27317 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:47:24.656088 27317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf: Bootstrap starting.
I20260926 13:47:24.657461 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 27248
I20260926 13:47:24.657857 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:39299
--local_ip_for_outbound_sockets=127.12.170.65
--tserver_master_addrs=127.12.170.126:35935
--webserver_port=35041
--webserver_interface=127.12.170.65
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:47:24.677448 27317 log.cc:824] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf: Log is configured to *not* fsync() on all Append() calls
I20260926 13:47:24.688072 27317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:47:24.688464 27317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf: Bootstrap complete.
I20260926 13:47:24.693732 27317 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ba174fc11a4dedafafffe0a841a7bf" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 35935 } }
I20260926 13:47:24.694239 27317 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ba174fc11a4dedafafffe0a841a7bf, State: Initialized, Role: FOLLOWER
I20260926 13:47:24.694777 27317 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ba174fc11a4dedafafffe0a841a7bf" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 35935 } }
I20260926 13:47:24.694916 27317 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:47:24.694983 27317 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:47:24.695082 27317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [term 1 FOLLOWER]: Advancing to term 2
I20260926 13:47:24.697484 27317 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ba174fc11a4dedafafffe0a841a7bf" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 35935 } }
I20260926 13:47:24.697853 27317 leader_election.cc:302] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8ba174fc11a4dedafafffe0a841a7bf; no voters:
I20260926 13:47:24.698220 27317 leader_election.cc:288] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 13:47:24.698328 27321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [term 2 FOLLOWER]: Leader election won for term 2
I20260926 13:47:24.698585 27321 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [term 2 LEADER]: Becoming Leader. State: Replica: b8ba174fc11a4dedafafffe0a841a7bf, State: Running, Role: LEADER
I20260926 13:47:24.698889 27321 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ba174fc11a4dedafafffe0a841a7bf" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 35935 } }
I20260926 13:47:24.699519 27317 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:47:24.700145 27322 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b8ba174fc11a4dedafafffe0a841a7bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ba174fc11a4dedafafffe0a841a7bf" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 35935 } } }
I20260926 13:47:24.700256 27322 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [sys.catalog]: This master's current role is: LEADER
I20260926 13:47:24.700637 27323 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8ba174fc11a4dedafafffe0a841a7bf. Latest consensus state: current_term: 2 leader_uuid: "b8ba174fc11a4dedafafffe0a841a7bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ba174fc11a4dedafafffe0a841a7bf" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 35935 } } }
I20260926 13:47:24.700726 27323 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [sys.catalog]: This master's current role is: LEADER
I20260926 13:47:24.701002 27327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:47:24.705561 27327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:47:24.706248 27327 catalog_manager.cc:1259] Loaded cluster ID: 92de00588af14ed29be18c09f59da2d0
I20260926 13:47:24.706293 27327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:47:24.707348 27327 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 13:47:24.710171 27327 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 13:47:24.714524 27327 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:47:24.715129 27327 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf: Loaded TSK: 0
I20260926 13:47:24.716132 27327 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 13:47:24.791242 27319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:24.791491 27319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:24.791532 27319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:24.795535 27319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:24.795598 27319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:24.795676 27319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:47:24.800920 27319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:39299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=35041
--tserver_master_addrs=127.12.170.126:35935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:24.801956 27319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:24.802920 27319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:47:24.812048 27345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:24.812036 27344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:24.812062 27347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:24.812355 27319 server_base.cc:1034] running on GCE node
I20260926 13:47:24.812726 27319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:24.813306 27319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:24.814460 27319 hybrid_clock.cc:648] HybridClock initialized: now 1790430444814436 us; error 41 us; skew 500 ppm
Sep 26 13:47:24 dist-test-slave-hc1k krb5kdc[26701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430444, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:24.817129 27319 init.cc:377] Logged in from keytab as kudu/127.12.170.65@KRBTEST.COM (short username kudu)
I20260926 13:47:24.817958 27319 webserver.cc:460] Webserver started at http://127.12.170.65:35041/ using document root <none> and password file <none>
I20260926 13:47:24.818526 27319 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:24.818596 27319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:47:24.822352 27319 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:47:24.824473 27354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:24.825702 27319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:47:24.825801 27319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "238606c7eb7c42fdb459e36d055dd84b"
format_stamp: "Formatted at 2026-09-26 13:47:23 on dist-test-slave-hc1k"
server_key: "dd88941eacd99e197e057daa65c20892"
server_key_iv: "8919bcd94b13eaeabce221940b6f60bb"
server_key_version: "encryptionkey@0"
I20260926 13:47:24.826292 27319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:24.849327 27319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:24.852277 27319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:24.852440 27319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:24.852981 27319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:47:24.853957 27361 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 13:47:24.857903 27319 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 13:47:24.857966 27319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260926 13:47:24.858028 27319 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 13:47:24.859607 27319 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 13:47:24.859678 27319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260926 13:47:24.859987 27361 tablet_bootstrap.cc:492] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b: Bootstrap starting.
I20260926 13:47:24.868789 27319 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:39299
I20260926 13:47:24.868846 27468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:39299 every 8 connection(s)
I20260926 13:47:24.869688 27319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:47:24.874737 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 27319
I20260926 13:47:24.875100 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:40559
--local_ip_for_outbound_sockets=127.12.170.66
--tserver_master_addrs=127.12.170.126:35935
--webserver_port=33833
--webserver_interface=127.12.170.66
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:47:24.878517 27361 log.cc:824] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b: Log is configured to *not* fsync() on all Append() calls
Sep 26 13:47:24 dist-test-slave-hc1k krb5kdc[26701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430444, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:47:24.886190 27361 tablet_bootstrap.cc:492] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:47:24.886587 27361 tablet_bootstrap.cc:492] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b: Bootstrap complete.
I20260926 13:47:24.887264 27472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:24.871553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.65:35763)
0926 13:47:24.872070 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:24.872144 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:24.873243 (+ 1099us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:24.874323 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:24.874332 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:24.874858 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:24.875550 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:24.875569 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.877103 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.877106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:24.877944 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:24.877953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.878846 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.878849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:24.879034 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:24.879706 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:24.879725 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:24.881546 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:24.883920 (+ 2374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:24.883926 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:24.883939 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:24.884291 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:24.884673 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:24.884676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:24.884678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:24.884868 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:24.885507 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:24.885514 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:24.885799 (+ 285us) client_negotiation.cc:769] Sending connection context
0926 13:47:24.885978 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 13:47:24.886296 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":116,"threads_started":1}
I20260926 13:47:24.887599 27471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:24.871622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:35763 (local address 127.12.170.126:35935)
0926 13:47:24.872005 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:24.872025 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:24.872278 (+ 253us) server_negotiation.cc:407] Connection header received
0926 13:47:24.873640 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:24.873651 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:24.873937 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:24.874198 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:24.875748 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.876959 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.878067 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.878694 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.878837 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:24.881755 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:24.881761 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:24.881776 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:24.881807 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:24.883786 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:24.884422 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:24.884428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:24.884433 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:24.884577 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:24.884970 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:24.884973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:24.884975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:24.885393 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:24.885516 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:24.886134 (+ 618us) server_negotiation.cc:299] Negotiation successful
0926 13:47:24.886515 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":97,"threads_started":1}
I20260926 13:47:24.887869 27361 ts_tablet_manager.cc:1397] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b: Time spent bootstrapping tablet: real 0.028s user 0.021s sys 0.006s
I20260926 13:47:24.890568 27469 heartbeater.cc:347] Connected to a master server at 127.12.170.126:35935
I20260926 13:47:24.890337 27361 raft_consensus.cc:348] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } } peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } }
I20260926 13:47:24.890797 27469 heartbeater.cc:464] Registering TS with master...
I20260926 13:47:24.890830 27361 raft_consensus.cc:729] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 238606c7eb7c42fdb459e36d055dd84b, State: Initialized, Role: FOLLOWER
I20260926 13:47:24.891304 27469 heartbeater.cc:511] Master 127.12.170.126:35935 requested a full tablet report, sending...
I20260926 13:47:24.891266 27361 consensus_queue.cc:260] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } } peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } }
I20260926 13:47:24.892221 27361 ts_tablet_manager.cc:1428] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260926 13:47:24.893847 27281 ts_manager.cc:194] Registered new tserver with Master: 238606c7eb7c42fdb459e36d055dd84b (127.12.170.65:39299)
I20260926 13:47:24.896938 27281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.65@KRBTEST.COM'} at 127.12.170.65:35763
I20260926 13:47:24.899267 27469 heartbeater.cc:503] Master 127.12.170.126:35935 was elected leader, sending a full tablet report...
Sep 26 13:47:24 dist-test-slave-hc1k krb5kdc[26701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430444, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for kudu/127.12.170.65@KRBTEST.COM
I20260926 13:47:24.907299 27476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:24.896428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:39299 (local address 127.0.0.1:59332)
0926 13:47:24.896952 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:24.896994 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:24.897149 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:24.897744 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:24.897750 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:24.898072 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:24.898165 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:24.898173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.900664 (+ 2491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.900686 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:24.901370 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:24.901379 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.902086 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.902088 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:24.902162 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:24.902644 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:24.902655 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:24.904055 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:24.905909 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:24.905916 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:24.905920 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:24.906120 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:24.906383 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:24.906385 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:24.906386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:24.906453 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:24.906866 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:24.906870 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:24.906954 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 13:47:24.907052 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 13:47:24.907182 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":107,"threads_started":1}
I20260926 13:47:24.907444 27477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:24.896509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59332 (local address 127.12.170.65:39299)
0926 13:47:24.897189 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:24.897200 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:24.897225 (+ 25us) server_negotiation.cc:407] Connection header received
0926 13:47:24.897290 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:24.897294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:24.897479 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:24.897734 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:24.899441 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.900544 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.901476 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:24.901972 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:24.902077 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:24.904183 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:24.904193 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:24.904206 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:24.904233 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:24.905814 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:24.906214 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:24.906219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:24.906221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:24.906308 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:24.906525 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:24.906528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:24.906529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:24.906779 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:24.906896 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:24.907145 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 13:47:24.907297 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":84,"threads_started":1}
I20260926 13:47:24.908324 27403 tablet_service.cc:1514] Processing DeleteTablet for tablet 5414033abcf94fdea69f21ba91630a43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:47:24 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:59332
I20260926 13:47:24.909060 27478 tablet_replica.cc:330] stopping tablet replica
I20260926 13:47:24.909371 27478 raft_consensus.cc:2229] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:47:24.909600 27478 raft_consensus.cc:2258] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:47:24.910480 27478 ts_tablet_manager.cc:1905] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:47:24.913774 27478 ts_tablet_manager.cc:1918] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:47:24.913849 27478 log.cc:1196] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/wal/wals/5414033abcf94fdea69f21ba91630a43
I20260926 13:47:24.914050 27478 ts_tablet_manager.cc:1939] T 5414033abcf94fdea69f21ba91630a43 P 238606c7eb7c42fdb459e36d055dd84b: Deleting consensus metadata
I20260926 13:47:24.915545 27266 catalog_manager.cc:4841] TS 238606c7eb7c42fdb459e36d055dd84b (127.12.170.65:39299): tablet 5414033abcf94fdea69f21ba91630a43 (table test-table [id=2bc2c0edc96148da88eb6176b8c08436]) successfully deleted
W20260926 13:47:24.987259 27473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:24.987498 27473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:24.987555 27473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:24.991456 27473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:24.991505 27473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:24.991585 27473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:47:24.996531 27473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:40559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=33833
--tserver_master_addrs=127.12.170.126:35935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:24.997448 27473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:24.998374 27473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:47:25.007493 27484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:25.007452 27486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:25.007447 27483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:25.007694 27473 server_base.cc:1034] running on GCE node
I20260926 13:47:25.008096 27473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:25.008706 27473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:25.009881 27473 hybrid_clock.cc:648] HybridClock initialized: now 1790430445009830 us; error 64 us; skew 500 ppm
Sep 26 13:47:25 dist-test-slave-hc1k krb5kdc[26701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430445, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:25.012583 27473 init.cc:377] Logged in from keytab as kudu/127.12.170.66@KRBTEST.COM (short username kudu)
I20260926 13:47:25.013399 27473 webserver.cc:460] Webserver started at http://127.12.170.66:33833/ using document root <none> and password file <none>
I20260926 13:47:25.013914 27473 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:25.013999 27473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:47:25.017900 27473 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:47:25.020016 27493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:25.021310 27473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:47:25.021433 27473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "3aee7b81806b4d9ebaff29a885d69ca9"
format_stamp: "Formatted at 2026-09-26 13:47:23 on dist-test-slave-hc1k"
server_key: "c63550c4996d4158616a004915edb31e"
server_key_iv: "6c02efe2825ecadcaf72ceb821b45515"
server_key_version: "encryptionkey@0"
I20260926 13:47:25.021957 27473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:25.042762 27473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:25.045778 27473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:25.045950 27473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:25.046566 27473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:47:25.047356 27473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:47:25.047401 27473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:25.047453 27473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:47:25.047482 27473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:25.055783 27473 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:40559
I20260926 13:47:25.055804 27606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:40559 every 8 connection(s)
I20260926 13:47:25.056762 27473 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:47:25.062129 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 27473
I20260926 13:47:25.062480 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:36135
--local_ip_for_outbound_sockets=127.12.170.67
--tserver_master_addrs=127.12.170.126:35935
--webserver_port=41331
--webserver_interface=127.12.170.67
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:47:25 dist-test-slave-hc1k krb5kdc[26701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430445, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:47:25.070187 27471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:25.058449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:40045 (local address 127.12.170.126:35935)
0926 13:47:25.058587 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:25.058591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:25.059395 (+ 804us) server_negotiation.cc:407] Connection header received
0926 13:47:25.060711 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:25.060714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:25.060774 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:25.060847 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:25.061991 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.062675 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.064198 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.064431 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.064522 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:25.066961 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:25.066964 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:25.066967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:25.066991 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:25.068342 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:25.068792 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:25.068795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:25.068796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:25.068870 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:25.069170 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:25.069173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:25.069175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:25.069340 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:25.069424 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:25.069877 (+ 453us) server_negotiation.cc:299] Negotiation successful
0926 13:47:25.070012 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:47:25.070914 27609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:25.058817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.66:40045)
0926 13:47:25.059253 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:25.059292 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:25.060502 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:25.061023 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:25.061030 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:25.061379 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:25.061827 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:25.061841 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.062818 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.062821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:25.064085 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:25.064095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.064568 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.064573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:25.064748 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:25.065279 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:25.065295 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:25.066805 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:25.068445 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:25.068451 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:25.068465 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:25.068702 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:25.068949 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:25.068952 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:25.068954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:25.069088 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:25.069436 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:25.069442 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:25.069666 (+ 224us) client_negotiation.cc:769] Sending connection context
0926 13:47:25.069821 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 13:47:25.070131 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":105,"threads_started":1}
I20260926 13:47:25.072013 27607 heartbeater.cc:347] Connected to a master server at 127.12.170.126:35935
I20260926 13:47:25.072247 27607 heartbeater.cc:464] Registering TS with master...
I20260926 13:47:25.072685 27607 heartbeater.cc:511] Master 127.12.170.126:35935 requested a full tablet report, sending...
I20260926 13:47:25.073622 27281 ts_manager.cc:194] Registered new tserver with Master: 3aee7b81806b4d9ebaff29a885d69ca9 (127.12.170.66:40559)
I20260926 13:47:25.074178 27281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.66@KRBTEST.COM'} at 127.12.170.66:40045
W20260926 13:47:25.168295 27610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:25.168519 27610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:25.168557 27610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:25.172489 27610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:25.172541 27610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:25.172602 27610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:47:25.177573 27610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:36135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=41331
--tserver_master_addrs=127.12.170.126:35935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:25.178506 27610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:25.179466 27610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:47:25.188215 27616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:25.188313 27618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:25.188349 27615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:25.188328 27610 server_base.cc:1034] running on GCE node
I20260926 13:47:25.188796 27610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:25.189388 27610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:25.190541 27610 hybrid_clock.cc:648] HybridClock initialized: now 1790430445190531 us; error 25 us; skew 500 ppm
Sep 26 13:47:25 dist-test-slave-hc1k krb5kdc[26701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430445, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:25.193184 27610 init.cc:377] Logged in from keytab as kudu/127.12.170.67@KRBTEST.COM (short username kudu)
I20260926 13:47:25.193945 27610 webserver.cc:460] Webserver started at http://127.12.170.67:41331/ using document root <none> and password file <none>
I20260926 13:47:25.194440 27610 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:25.194504 27610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:47:25.198096 27610 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:47:25.200110 27625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:25.201363 27610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:47:25.201447 27610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "7466e77c905546d8a1be7fadd7a5817b"
format_stamp: "Formatted at 2026-09-26 13:47:24 on dist-test-slave-hc1k"
server_key: "7545e64d9744de102e6566d9b29260b8"
server_key_iv: "c5abd7297a1f759f7d65dc25df72f6e3"
server_key_version: "encryptionkey@0"
I20260926 13:47:25.201964 27610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:25.214517 27610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:25.217468 27610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:25.217597 27610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:25.218073 27610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:47:25.218786 27610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:47:25.218828 27610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:25.218865 27610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:47:25.218884 27610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:25.226890 27610 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:36135
I20260926 13:47:25.226949 27738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:36135 every 8 connection(s)
I20260926 13:47:25.227828 27610 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:47:25.228101 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 27610
I20260926 13:47:25.236912 27471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:25.228784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47954 (local address 127.12.170.126:35935)
0926 13:47:25.228932 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:25.228936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:25.229270 (+ 334us) server_negotiation.cc:407] Connection header received
0926 13:47:25.229452 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:25.229456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:25.229511 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:25.229594 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:25.230400 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.231175 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.232404 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.232668 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.232741 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:25.234115 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:25.234119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:25.234122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:25.234147 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:25.235449 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:25.235865 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:25.235868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:25.235870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:25.235959 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:25.236211 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:25.236213 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:25.236214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:25.236356 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:25.236531 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:25.236695 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 13:47:25.236787 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 13:47:25 dist-test-slave-hc1k krb5kdc[26701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430445, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:47:25.241268 27742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:25.229703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:45937 (local address 127.12.170.126:35935)
0926 13:47:25.229928 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:25.229932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:25.230759 (+ 827us) server_negotiation.cc:407] Connection header received
0926 13:47:25.232296 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:25.232299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:25.232338 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:25.232417 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:25.233733 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.234317 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.235577 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.235870 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.235963 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:25.238380 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:25.238383 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:25.238386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:25.238409 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:25.239492 (+ 1083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:25.239956 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:25.239960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:25.239961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:25.240040 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:25.240331 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:25.240334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:25.240336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:25.240495 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:25.240575 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:25.241010 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 13:47:25.241119 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":86,"threads_started":1}
I20260926 13:47:25.241966 27743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:25.230190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.67:45937)
0926 13:47:25.230623 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:25.230679 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:25.232106 (+ 1427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:25.232558 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:25.232565 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:25.232977 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:25.233574 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:25.233594 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.234434 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.234437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:25.235469 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:25.235478 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.235996 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.236000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:25.236213 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:25.236723 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:25.236741 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:25.238237 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:25.239594 (+ 1357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:25.239599 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:25.239611 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:25.239870 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:25.240123 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:25.240126 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:25.240128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:25.240250 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:25.240585 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:25.240591 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:25.240804 (+ 213us) client_negotiation.cc:769] Sending connection context
0926 13:47:25.240956 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 13:47:25.241247 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":112,"threads_started":1}
I20260926 13:47:25.242861 27739 heartbeater.cc:347] Connected to a master server at 127.12.170.126:35935
I20260926 13:47:25.243053 27739 heartbeater.cc:464] Registering TS with master...
I20260926 13:47:25.243479 27739 heartbeater.cc:511] Master 127.12.170.126:35935 requested a full tablet report, sending...
I20260926 13:47:25.244372 27281 ts_manager.cc:194] Registered new tserver with Master: 7466e77c905546d8a1be7fadd7a5817b (127.12.170.67:36135)
I20260926 13:47:25.244864 27281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.67@KRBTEST.COM'} at 127.12.170.67:45937
I20260926 13:47:25.249799 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:47:25.253624 27742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:25.250315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47958 (local address 127.12.170.126:35935)
0926 13:47:25.250411 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:25.250415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:25.250581 (+ 166us) server_negotiation.cc:407] Connection header received
0926 13:47:25.250718 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:25.250720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:25.250802 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:25.250882 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:47:25.251097 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.251559 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.252399 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.252617 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.252702 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:25.252971 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:47:25.253292 (+ 321us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:47:25.253388 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:25.253456 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 13:47:25.253507 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 13:47:25.254146 27281 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:47958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:47:25.257164 27281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:47:25.264815 27403 tablet_service.cc:1467] Processing CreateTablet for tablet 895e6e19add44fc7884276270cf8417a (DEFAULT_TABLE table=test-table [id=d777abe8121d4b499c537be2ace37185]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:47:25.265467 27403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 895e6e19add44fc7884276270cf8417a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:47:25.268321 27361 tablet_bootstrap.cc:492] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b: Bootstrap starting.
I20260926 13:47:25.268925 27476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:25.263411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:36135 (local address 127.0.0.1:55854)
0926 13:47:25.263541 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:25.263557 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:25.263637 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:25.264343 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:25.264346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:25.264362 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:47:25.264461 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:25.264467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.266101 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.266105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:25.267258 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:25.267270 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.268438 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.268441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:25.268619 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:25.268658 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 13:47:25.268723 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 13:47:25.268799 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":35}
I20260926 13:47:25.269127 27748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:25.263582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55854 (local address 127.12.170.67:36135)
0926 13:47:25.263868 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:25.263878 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:25.263897 (+ 19us) server_negotiation.cc:407] Connection header received
0926 13:47:25.263966 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:25.263971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:25.264210 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:25.264371 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:47:25.264580 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.265943 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.267375 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.268317 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.268600 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:25.268852 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:25.268900 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 13:47:25.268982 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":123,"threads_started":1}
I20260926 13:47:25.269122 27749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:25.263869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:40559 (local address 127.0.0.1:51118)
0926 13:47:25.264119 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:25.264133 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:25.264216 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:25.265042 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:25.265045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:25.265062 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:47:25.265142 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:25.265148 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.266369 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.266372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:25.267189 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:25.267198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.268438 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.268441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:25.268797 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:25.268824 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 13:47:25.268899 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 13:47:25.268967 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":255,"thread_start_us":110,"threads_started":1}
I20260926 13:47:25.269229 27361 tablet_bootstrap.cc:654] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b: Neither blocks nor log segments found. Creating new log.
I20260926 13:47:25.269631 27750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:25.264033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51118 (local address 127.12.170.66:40559)
0926 13:47:25.264413 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:25.264423 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:25.264443 (+ 20us) server_negotiation.cc:407] Connection header received
0926 13:47:25.264526 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:25.264532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:25.264735 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:25.264883 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:47:25.265257 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.266260 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.267299 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.268145 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.268243 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:25.268384 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:25.269355 (+ 971us) server_negotiation.cc:299] Negotiation successful
0926 13:47:25.269459 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":86,"threads_started":1}
I20260926 13:47:25.271328 27361 tablet_bootstrap.cc:492] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b: No bootstrap required, opened a new log
I20260926 13:47:25.271427 27361 ts_tablet_manager.cc:1397] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 13:47:25.271616 27673 tablet_service.cc:1467] Processing CreateTablet for tablet 895e6e19add44fc7884276270cf8417a (DEFAULT_TABLE table=test-table [id=d777abe8121d4b499c537be2ace37185]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:47:25.271965 27361 raft_consensus.cc:348] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } } peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } }
I20260926 13:47:25.272104 27361 raft_consensus.cc:374] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:47:25.272152 27361 raft_consensus.cc:729] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 238606c7eb7c42fdb459e36d055dd84b, State: Initialized, Role: FOLLOWER
I20260926 13:47:25.272289 27361 consensus_queue.cc:260] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } } peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } }
I20260926 13:47:25.272612 27673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 895e6e19add44fc7884276270cf8417a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:47:25.272701 27361 ts_tablet_manager.cc:1428] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 13:47:25.272585 27541 tablet_service.cc:1467] Processing CreateTablet for tablet 895e6e19add44fc7884276270cf8417a (DEFAULT_TABLE table=test-table [id=d777abe8121d4b499c537be2ace37185]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:47:25.273566 27541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 895e6e19add44fc7884276270cf8417a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:47:25.279409 27752 tablet_bootstrap.cc:492] T 895e6e19add44fc7884276270cf8417a P 7466e77c905546d8a1be7fadd7a5817b: Bootstrap starting.
I20260926 13:47:25.279587 27753 tablet_bootstrap.cc:492] T 895e6e19add44fc7884276270cf8417a P 3aee7b81806b4d9ebaff29a885d69ca9: Bootstrap starting.
I20260926 13:47:25.281548 27752 tablet_bootstrap.cc:654] T 895e6e19add44fc7884276270cf8417a P 7466e77c905546d8a1be7fadd7a5817b: Neither blocks nor log segments found. Creating new log.
I20260926 13:47:25.281736 27753 tablet_bootstrap.cc:654] T 895e6e19add44fc7884276270cf8417a P 3aee7b81806b4d9ebaff29a885d69ca9: Neither blocks nor log segments found. Creating new log.
I20260926 13:47:25.282275 27752 log.cc:824] T 895e6e19add44fc7884276270cf8417a P 7466e77c905546d8a1be7fadd7a5817b: Log is configured to *not* fsync() on all Append() calls
I20260926 13:47:25.282686 27753 log.cc:824] T 895e6e19add44fc7884276270cf8417a P 3aee7b81806b4d9ebaff29a885d69ca9: Log is configured to *not* fsync() on all Append() calls
I20260926 13:47:25.284106 27752 tablet_bootstrap.cc:492] T 895e6e19add44fc7884276270cf8417a P 7466e77c905546d8a1be7fadd7a5817b: No bootstrap required, opened a new log
I20260926 13:47:25.284266 27752 ts_tablet_manager.cc:1397] T 895e6e19add44fc7884276270cf8417a P 7466e77c905546d8a1be7fadd7a5817b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 13:47:25.284492 27753 tablet_bootstrap.cc:492] T 895e6e19add44fc7884276270cf8417a P 3aee7b81806b4d9ebaff29a885d69ca9: No bootstrap required, opened a new log
I20260926 13:47:25.284657 27753 ts_tablet_manager.cc:1397] T 895e6e19add44fc7884276270cf8417a P 3aee7b81806b4d9ebaff29a885d69ca9: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260926 13:47:25.286921 27752 raft_consensus.cc:348] T 895e6e19add44fc7884276270cf8417a P 7466e77c905546d8a1be7fadd7a5817b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } } peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } }
I20260926 13:47:25.287129 27752 raft_consensus.cc:374] T 895e6e19add44fc7884276270cf8417a P 7466e77c905546d8a1be7fadd7a5817b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:47:25.287189 27752 raft_consensus.cc:729] T 895e6e19add44fc7884276270cf8417a P 7466e77c905546d8a1be7fadd7a5817b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7466e77c905546d8a1be7fadd7a5817b, State: Initialized, Role: FOLLOWER
I20260926 13:47:25.287395 27753 raft_consensus.cc:348] T 895e6e19add44fc7884276270cf8417a P 3aee7b81806b4d9ebaff29a885d69ca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } } peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } }
I20260926 13:47:25.287678 27753 raft_consensus.cc:374] T 895e6e19add44fc7884276270cf8417a P 3aee7b81806b4d9ebaff29a885d69ca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:47:25.287617 27752 consensus_queue.cc:260] T 895e6e19add44fc7884276270cf8417a P 7466e77c905546d8a1be7fadd7a5817b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } } peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } }
I20260926 13:47:25.287743 27753 raft_consensus.cc:729] T 895e6e19add44fc7884276270cf8417a P 3aee7b81806b4d9ebaff29a885d69ca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aee7b81806b4d9ebaff29a885d69ca9, State: Initialized, Role: FOLLOWER
I20260926 13:47:25.288172 27753 consensus_queue.cc:260] T 895e6e19add44fc7884276270cf8417a P 3aee7b81806b4d9ebaff29a885d69ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } } peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } }
I20260926 13:47:25.288369 27739 heartbeater.cc:503] Master 127.12.170.126:35935 was elected leader, sending a full tablet report...
I20260926 13:47:25.288616 27752 ts_tablet_manager.cc:1428] T 895e6e19add44fc7884276270cf8417a P 7466e77c905546d8a1be7fadd7a5817b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 13:47:25.289125 27753 ts_tablet_manager.cc:1428] T 895e6e19add44fc7884276270cf8417a P 3aee7b81806b4d9ebaff29a885d69ca9: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260926 13:47:25.289458 27607 heartbeater.cc:503] Master 127.12.170.126:35935 was elected leader, sending a full tablet report...
W20260926 13:47:25.307677 27608 tablet.cc:2369] T 895e6e19add44fc7884276270cf8417a P 3aee7b81806b4d9ebaff29a885d69ca9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:47:25.370888 27470 tablet.cc:2369] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:47:25.479415 27740 tablet.cc:2369] T 895e6e19add44fc7884276270cf8417a P 7466e77c905546d8a1be7fadd7a5817b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:47:25.569267 27475 raft_consensus.cc:482] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:47:25.569481 27475 raft_consensus.cc:504] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } } peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } }
I20260926 13:47:25.570528 27475 leader_election.cc:288] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7466e77c905546d8a1be7fadd7a5817b (127.12.170.67:36135), 3aee7b81806b4d9ebaff29a885d69ca9 (127.12.170.66:40559)
I20260926 13:47:25.574307 27750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:25.571139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:45687 (local address 127.12.170.66:40559)
0926 13:47:25.571257 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:25.571262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:25.571275 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:47:25.571327 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:25.571330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:25.571379 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:25.571451 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:47:25.571874 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.572432 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.573366 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.573742 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.573840 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:25.573890 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:25.574107 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 13:47:25.574173 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 13:47:25.574314 27748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:25.570963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:34215 (local address 127.12.170.67:36135)
0926 13:47:25.571084 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:25.571089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:25.571261 (+ 172us) server_negotiation.cc:407] Connection header received
0926 13:47:25.571373 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:25.571375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:25.571414 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:25.571481 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:47:25.571735 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.572294 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.573350 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.573741 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.573840 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:25.573890 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:25.574115 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 13:47:25.574185 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 13:47:25.574324 27758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:25.570743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:40559 (local address 127.12.170.65:45687)
0926 13:47:25.571001 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:25.571016 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:25.571142 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:25.571634 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:25.571636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:25.571677 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:47:25.571770 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:25.571776 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.572533 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.572536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:25.573255 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:25.573262 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.573842 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.573845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:25.573961 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:25.574013 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 13:47:25.574082 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 13:47:25.574193 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":43,"thread_start_us":88,"threads_started":1}
I20260926 13:47:25.574323 27759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:25.570743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:36135 (local address 127.12.170.65:34215)
0926 13:47:25.571169 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:25.571183 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:25.571293 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:25.571500 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:25.571503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:25.571519 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:47:25.571630 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:25.571635 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.572386 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.572389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:25.573231 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:25.573240 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.573841 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.573845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:25.573998 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:25.574021 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 13:47:25.574084 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 13:47:25.574184 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":76,"thread_start_us":79,"threads_started":1}
I20260926 13:47:25.574996 27561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895e6e19add44fc7884276270cf8417a" candidate_uuid: "238606c7eb7c42fdb459e36d055dd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" is_pre_election: true
I20260926 13:47:25.575130 27693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895e6e19add44fc7884276270cf8417a" candidate_uuid: "238606c7eb7c42fdb459e36d055dd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7466e77c905546d8a1be7fadd7a5817b" is_pre_election: true
I20260926 13:47:25.575299 27561 raft_consensus.cc:2454] T 895e6e19add44fc7884276270cf8417a P 3aee7b81806b4d9ebaff29a885d69ca9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 238606c7eb7c42fdb459e36d055dd84b in term 0.
I20260926 13:47:25.575405 27693 raft_consensus.cc:2454] T 895e6e19add44fc7884276270cf8417a P 7466e77c905546d8a1be7fadd7a5817b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 238606c7eb7c42fdb459e36d055dd84b in term 0.
I20260926 13:47:25.575731 27356 leader_election.cc:302] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 238606c7eb7c42fdb459e36d055dd84b, 3aee7b81806b4d9ebaff29a885d69ca9; no voters:
I20260926 13:47:25.575981 27475 raft_consensus.cc:2781] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:47:25.576083 27475 raft_consensus.cc:482] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:47:25.576115 27475 raft_consensus.cc:3037] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:47:25.577071 27475 raft_consensus.cc:504] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } } peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } }
I20260926 13:47:25.577387 27475 leader_election.cc:288] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [CANDIDATE]: Term 1 election: Requested vote from peers 7466e77c905546d8a1be7fadd7a5817b (127.12.170.67:36135), 3aee7b81806b4d9ebaff29a885d69ca9 (127.12.170.66:40559)
I20260926 13:47:25.577708 27561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895e6e19add44fc7884276270cf8417a" candidate_uuid: "238606c7eb7c42fdb459e36d055dd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aee7b81806b4d9ebaff29a885d69ca9"
I20260926 13:47:25.577708 27693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "895e6e19add44fc7884276270cf8417a" candidate_uuid: "238606c7eb7c42fdb459e36d055dd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7466e77c905546d8a1be7fadd7a5817b"
I20260926 13:47:25.577838 27693 raft_consensus.cc:3037] T 895e6e19add44fc7884276270cf8417a P 7466e77c905546d8a1be7fadd7a5817b [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:47:25.577838 27561 raft_consensus.cc:3037] T 895e6e19add44fc7884276270cf8417a P 3aee7b81806b4d9ebaff29a885d69ca9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:47:25.579078 27561 raft_consensus.cc:2454] T 895e6e19add44fc7884276270cf8417a P 3aee7b81806b4d9ebaff29a885d69ca9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 238606c7eb7c42fdb459e36d055dd84b in term 1.
I20260926 13:47:25.579078 27693 raft_consensus.cc:2454] T 895e6e19add44fc7884276270cf8417a P 7466e77c905546d8a1be7fadd7a5817b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 238606c7eb7c42fdb459e36d055dd84b in term 1.
I20260926 13:47:25.579324 27356 leader_election.cc:302] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 238606c7eb7c42fdb459e36d055dd84b, 3aee7b81806b4d9ebaff29a885d69ca9; no voters:
I20260926 13:47:25.579480 27475 raft_consensus.cc:2781] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:47:25.579684 27475 raft_consensus.cc:686] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 1 LEADER]: Becoming Leader. State: Replica: 238606c7eb7c42fdb459e36d055dd84b, State: Running, Role: LEADER
I20260926 13:47:25.579921 27475 consensus_queue.cc:237] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } } peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } }
I20260926 13:47:25.582546 27281 catalog_manager.cc:5495] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b reported cstate change: term changed from 0 to 1, leader changed from <none> to 238606c7eb7c42fdb459e36d055dd84b (127.12.170.65). New cstate: current_term: 1 leader_uuid: "238606c7eb7c42fdb459e36d055dd84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } health_report { overall_health: HEALTHY } } }
I20260926 13:47:25.633116 27763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:25.629609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59336 (local address 127.12.170.65:39299)
0926 13:47:25.629860 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:25.629873 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:25.629885 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:47:25.629956 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:25.629958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:25.630009 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:25.630128 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:47:25.630283 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.630867 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.631812 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:25.632078 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:25.632155 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:25.632360 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:47:25.632775 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:47:25.632855 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:25.632919 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 13:47:25.632967 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":104,"threads_started":1}
I20260926 13:47:25.636795 27561 raft_consensus.cc:1261] T 895e6e19add44fc7884276270cf8417a P 3aee7b81806b4d9ebaff29a885d69ca9 [term 1 FOLLOWER]: Refusing update from remote peer 238606c7eb7c42fdb459e36d055dd84b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:47:25.636795 27693 raft_consensus.cc:1261] T 895e6e19add44fc7884276270cf8417a P 7466e77c905546d8a1be7fadd7a5817b [term 1 FOLLOWER]: Refusing update from remote peer 238606c7eb7c42fdb459e36d055dd84b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:47:25.637570 27760 consensus_queue.cc:1035] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:47:25.637753 27475 consensus_queue.cc:1035] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:47:25.646183 27765 mvcc.cc:205] Tried to move back new op lower bound from 7333603105322049536 to 7333603105097441280. Current Snapshot: MvccSnapshot[applied={T|T < 7333603105322049536}]
I20260926 13:47:25.654075 27280 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:47958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:47:25.654217 27280 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:47958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:47:25.656212 27280 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf: Sending DeleteTablet for 3 replicas of tablet 895e6e19add44fc7884276270cf8417a
I20260926 13:47:25.656803 27673 tablet_service.cc:1514] Processing DeleteTablet for tablet 895e6e19add44fc7884276270cf8417a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:47:25 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:55854
I20260926 13:47:25.657133 27541 tablet_service.cc:1514] Processing DeleteTablet for tablet 895e6e19add44fc7884276270cf8417a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:47:25 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:51118
I20260926 13:47:25.657517 27775 tablet_replica.cc:330] T 895e6e19add44fc7884276270cf8417a P 3aee7b81806b4d9ebaff29a885d69ca9: stopping tablet replica
I20260926 13:47:25.657692 27403 tablet_service.cc:1514] Processing DeleteTablet for tablet 895e6e19add44fc7884276270cf8417a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:47:25 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:59332
I20260926 13:47:25.657809 27775 raft_consensus.cc:2229] T 895e6e19add44fc7884276270cf8417a P 3aee7b81806b4d9ebaff29a885d69ca9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:47:25.658115 27775 raft_consensus.cc:2258] T 895e6e19add44fc7884276270cf8417a P 3aee7b81806b4d9ebaff29a885d69ca9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:47:25.658269 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 27319
I20260926 13:47:25.659997 27775 ts_tablet_manager.cc:1905] T 895e6e19add44fc7884276270cf8417a P 3aee7b81806b4d9ebaff29a885d69ca9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:47:25.662060 27775 ts_tablet_manager.cc:1918] T 895e6e19add44fc7884276270cf8417a P 3aee7b81806b4d9ebaff29a885d69ca9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:47:25.662142 27775 log.cc:1196] T 895e6e19add44fc7884276270cf8417a P 3aee7b81806b4d9ebaff29a885d69ca9: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/wal/wals/895e6e19add44fc7884276270cf8417a
I20260926 13:47:25.662396 27775 ts_tablet_manager.cc:1939] T 895e6e19add44fc7884276270cf8417a P 3aee7b81806b4d9ebaff29a885d69ca9: Deleting consensus metadata
I20260926 13:47:25.663403 27266 catalog_manager.cc:4841] TS 3aee7b81806b4d9ebaff29a885d69ca9 (127.12.170.66:40559): tablet 895e6e19add44fc7884276270cf8417a (table test-table [id=d777abe8121d4b499c537be2ace37185]) successfully deleted
W20260926 13:47:25.664846 27266 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.170.65:39299 (error 108)
I20260926 13:47:25.665273 27774 tablet_replica.cc:330] T 895e6e19add44fc7884276270cf8417a P 7466e77c905546d8a1be7fadd7a5817b: stopping tablet replica
I20260926 13:47:25.665562 27774 raft_consensus.cc:2229] T 895e6e19add44fc7884276270cf8417a P 7466e77c905546d8a1be7fadd7a5817b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:47:25.665767 27749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:25.665525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:39299 (local address 127.0.0.1:59342)
0926 13:47:25.665637 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:25.665695 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.65:39299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 13:47:25.665855 27774 raft_consensus.cc:2258] T 895e6e19add44fc7884276270cf8417a P 7466e77c905546d8a1be7fadd7a5817b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 13:47:25.665889 27266 catalog_manager.cc:4584] TS 238606c7eb7c42fdb459e36d055dd84b (127.12.170.65:39299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 895e6e19add44fc7884276270cf8417a: Network error: Client connection negotiation failed: client connection to 127.12.170.65:39299: connect: Connection refused (error 111)
I20260926 13:47:25.666193 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 27473
I20260926 13:47:25.667450 27774 ts_tablet_manager.cc:1905] T 895e6e19add44fc7884276270cf8417a P 7466e77c905546d8a1be7fadd7a5817b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:47:25.669464 27774 ts_tablet_manager.cc:1918] T 895e6e19add44fc7884276270cf8417a P 7466e77c905546d8a1be7fadd7a5817b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:47:25.669559 27774 log.cc:1196] T 895e6e19add44fc7884276270cf8417a P 7466e77c905546d8a1be7fadd7a5817b: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/895e6e19add44fc7884276270cf8417a
I20260926 13:47:25.669806 27774 ts_tablet_manager.cc:1939] T 895e6e19add44fc7884276270cf8417a P 7466e77c905546d8a1be7fadd7a5817b: Deleting consensus metadata
I20260926 13:47:25.670795 27268 catalog_manager.cc:4841] TS 7466e77c905546d8a1be7fadd7a5817b (127.12.170.67:36135): tablet 895e6e19add44fc7884276270cf8417a (table test-table [id=d777abe8121d4b499c537be2ace37185]) successfully deleted
I20260926 13:47:25.671267 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 27610
I20260926 13:47:25.675812 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 27248
I20260926 13:47:25.680876 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:35935
--webserver_interface=127.12.170.126
--webserver_port=41015
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:35935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:47:25.784693 27778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:25.784945 27778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:25.784991 27778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:25.788872 27778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:47:25.788924 27778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:25.788941 27778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:25.788955 27778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:47:25.788969 27778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:47:25.793897 27778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:35935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:35935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=41015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:25.794799 27778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:25.795779 27778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:47:25.803633 27783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:25.803695 27778 server_base.cc:1034] running on GCE node
W20260926 13:47:25.803637 27786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:25.803633 27784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:25.804410 27778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:25.805445 27778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:25.806612 27778 hybrid_clock.cc:648] HybridClock initialized: now 1790430445806593 us; error 32 us; skew 500 ppm
Sep 26 13:47:25 dist-test-slave-hc1k krb5kdc[26701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430445, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:25.809136 27778 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:47:25.809916 27778 webserver.cc:460] Webserver started at http://127.12.170.126:41015/ using document root <none> and password file <none>
I20260926 13:47:25.810418 27778 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:25.810480 27778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:47:25.814098 27778 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:47:25.816094 27793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:25.817297 27778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:47:25.817408 27778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "b8ba174fc11a4dedafafffe0a841a7bf"
format_stamp: "Formatted at 2026-09-26 13:47:23 on dist-test-slave-hc1k"
server_key: "bd99a3848790e23f967ebec63b9432da"
server_key_iv: "ac49c734ef4f7d58693d8ef477b21fcb"
server_key_version: "encryptionkey@0"
I20260926 13:47:25.817893 27778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:25.830464 27778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:25.833055 27778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:25.833205 27778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:25.839778 27778 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:35935
I20260926 13:47:25.840039 27845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:35935 every 8 connection(s)
I20260926 13:47:25.840755 27778 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:47:25.844753 27846 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:47:25.845997 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 27778
I20260926 13:47:25.846282 27846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf: Bootstrap starting.
I20260926 13:47:25.846362 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:39299
--local_ip_for_outbound_sockets=127.12.170.65
--tserver_master_addrs=127.12.170.126:35935
--webserver_port=35041
--webserver_interface=127.12.170.65
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:47:25.868467 27846 log.cc:824] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf: Log is configured to *not* fsync() on all Append() calls
I20260926 13:47:25.882643 27846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:47:25.883062 27846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf: Bootstrap complete.
I20260926 13:47:25.888449 27846 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ba174fc11a4dedafafffe0a841a7bf" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 35935 } }
I20260926 13:47:25.888953 27846 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8ba174fc11a4dedafafffe0a841a7bf, State: Initialized, Role: FOLLOWER
I20260926 13:47:25.889536 27846 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ba174fc11a4dedafafffe0a841a7bf" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 35935 } }
I20260926 13:47:25.889693 27846 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:47:25.889760 27846 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:47:25.889851 27846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [term 2 FOLLOWER]: Advancing to term 3
I20260926 13:47:25.892371 27846 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ba174fc11a4dedafafffe0a841a7bf" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 35935 } }
I20260926 13:47:25.892750 27846 leader_election.cc:302] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8ba174fc11a4dedafafffe0a841a7bf; no voters:
I20260926 13:47:25.893123 27846 leader_election.cc:288] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 13:47:25.893242 27850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [term 3 FOLLOWER]: Leader election won for term 3
I20260926 13:47:25.893488 27850 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [term 3 LEADER]: Becoming Leader. State: Replica: b8ba174fc11a4dedafafffe0a841a7bf, State: Running, Role: LEADER
I20260926 13:47:25.893810 27850 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ba174fc11a4dedafafffe0a841a7bf" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 35935 } }
I20260926 13:47:25.894410 27846 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:47:25.895179 27852 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8ba174fc11a4dedafafffe0a841a7bf. Latest consensus state: current_term: 3 leader_uuid: "b8ba174fc11a4dedafafffe0a841a7bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ba174fc11a4dedafafffe0a841a7bf" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 35935 } } }
I20260926 13:47:25.895306 27852 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [sys.catalog]: This master's current role is: LEADER
I20260926 13:47:25.895161 27851 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b8ba174fc11a4dedafafffe0a841a7bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8ba174fc11a4dedafafffe0a841a7bf" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 35935 } } }
I20260926 13:47:25.895426 27851 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8ba174fc11a4dedafafffe0a841a7bf [sys.catalog]: This master's current role is: LEADER
I20260926 13:47:25.895962 27857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:47:25.900956 27857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:47:25.901618 27857 catalog_manager.cc:1259] Loaded cluster ID: 92de00588af14ed29be18c09f59da2d0
I20260926 13:47:25.901677 27857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 13:47:25.902661 27857 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9a58fb29ad google::LogMessage::Fail() at ??:0
@ 0x7f9a58fb588a google::LogMessage::SendToLog() at ??:0
@ 0x7f9a58fb24de google::LogMessage::Flush() at ??:0
@ 0x7f9a58fb3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9a638f5fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f9a638f6d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9a638f3579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f9a63937640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f9a5b2043d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9a5b204c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f9a5b2063f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f9a5b1f5512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9a5c8396db start_thread at ??:0
@ 0x7f9a57ab061f clone at ??:0
W20260926 13:47:25.984072 27848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:25.984329 27848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:25.984390 27848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:25.988350 27848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:25.988409 27848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:25.988483 27848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:47:25.993731 27848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:39299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=35041
--tserver_master_addrs=127.12.170.126:35935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:25.994779 27848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:25.995813 27848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:47:26.005273 27887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:26.005299 27888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:26.005273 27890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:26.005735 27848 server_base.cc:1034] running on GCE node
I20260926 13:47:26.006227 27848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:26.007035 27848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:26.008215 27848 hybrid_clock.cc:648] HybridClock initialized: now 1790430446008181 us; error 42 us; skew 500 ppm
Sep 26 13:47:26 dist-test-slave-hc1k krb5kdc[26701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430446, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:26.011508 27848 init.cc:377] Logged in from keytab as kudu/127.12.170.65@KRBTEST.COM (short username kudu)
I20260926 13:47:26.012686 27848 webserver.cc:460] Webserver started at http://127.12.170.65:35041/ using document root <none> and password file <none>
I20260926 13:47:26.013412 27848 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:26.013507 27848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:47:26.018855 27848 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:47:26.022279 27897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:26.023831 27848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:47:26.023950 27848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "238606c7eb7c42fdb459e36d055dd84b"
format_stamp: "Formatted at 2026-09-26 13:47:23 on dist-test-slave-hc1k"
server_key: "dd88941eacd99e197e057daa65c20892"
server_key_iv: "8919bcd94b13eaeabce221940b6f60bb"
server_key_version: "encryptionkey@0"
I20260926 13:47:26.024667 27848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:26.041280 27848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:26.044662 27848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:26.044862 27848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:26.045523 27848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:47:26.046882 27904 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 13:47:26.050952 27848 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 13:47:26.051024 27848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260926 13:47:26.051091 27848 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 13:47:26.053208 27848 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 13:47:26.053263 27848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260926 13:47:26.053563 27904 tablet_bootstrap.cc:492] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b: Bootstrap starting.
I20260926 13:47:26.063827 27848 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:39299
I20260926 13:47:26.064144 28011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:39299 every 8 connection(s)
I20260926 13:47:26.065158 27848 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:47:26.072358 27904 log.cc:824] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b: Log is configured to *not* fsync() on all Append() calls
I20260926 13:47:26.073418 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 27848
I20260926 13:47:26.073855 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:40559
--local_ip_for_outbound_sockets=127.12.170.66
--tserver_master_addrs=127.12.170.126:35935
--webserver_port=33833
--webserver_interface=127.12.170.66
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:47:26.079273 27904 tablet_bootstrap.cc:492] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:47:26.079571 27904 tablet_bootstrap.cc:492] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b: Bootstrap complete.
I20260926 13:47:26.080470 27904 ts_tablet_manager.cc:1397] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.008s
I20260926 13:47:26.083645 27904 raft_consensus.cc:348] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } } peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } }
I20260926 13:47:26.084206 27904 raft_consensus.cc:729] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 238606c7eb7c42fdb459e36d055dd84b, State: Initialized, Role: FOLLOWER
I20260926 13:47:26.084791 27904 consensus_queue.cc:260] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } } peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } }
I20260926 13:47:26.085822 27904 ts_tablet_manager.cc:1428] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260926 13:47:26.124699 28014 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:47:26.067463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.65:56919)
0926 13:47:26.068099 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:26.068156 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:26.069205 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:26.123924 (+ 54719us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":166,"threads_started":1}
W20260926 13:47:26.125310 28012 heartbeater.cc:644] Failed to heartbeat to 127.12.170.126:35935 (0 consecutive failures): Network error: Failed to ping master at 127.12.170.126:35935: Client connection negotiation failed: client connection to 127.12.170.126:35935: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 13:47:26.126058 28014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:26.125804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.65:58735)
0926 13:47:26.125931 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:26.125999 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 13:47:26.126708 28014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:26.126526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.65:34577)
0926 13:47:26.126619 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:26.126660 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 13:47:26.126855 28012 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:47:26.203883 28016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:26.204126 28016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:26.204188 28016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:26.208205 28016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:26.208264 28016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:26.208346 28016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:47:26.213514 28016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:40559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=33833
--tserver_master_addrs=127.12.170.126:35935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:26.214457 28016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:26.215423 28016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:47:26.224417 28023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:26.224429 28022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:26.224476 28025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:26.224663 28016 server_base.cc:1034] running on GCE node
I20260926 13:47:26.225088 28016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:26.225667 28016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:26.226814 28016 hybrid_clock.cc:648] HybridClock initialized: now 1790430446226791 us; error 40 us; skew 500 ppm
Sep 26 13:47:26 dist-test-slave-hc1k krb5kdc[26701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430446, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:26.229245 28016 init.cc:377] Logged in from keytab as kudu/127.12.170.66@KRBTEST.COM (short username kudu)
I20260926 13:47:26.230082 28016 webserver.cc:460] Webserver started at http://127.12.170.66:33833/ using document root <none> and password file <none>
I20260926 13:47:26.230638 28016 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:26.230705 28016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:47:26.234270 28016 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260926 13:47:26.236215 28032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:26.237422 28016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:47:26.237555 28016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "3aee7b81806b4d9ebaff29a885d69ca9"
format_stamp: "Formatted at 2026-09-26 13:47:23 on dist-test-slave-hc1k"
server_key: "c63550c4996d4158616a004915edb31e"
server_key_iv: "6c02efe2825ecadcaf72ceb821b45515"
server_key_version: "encryptionkey@0"
I20260926 13:47:26.238085 28016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:26.253057 28016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:26.255823 28016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:26.255993 28016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:26.256529 28016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:47:26.257299 28016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:47:26.257345 28016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:26.257397 28016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:47:26.257426 28016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:26.265962 28016 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:40559
I20260926 13:47:26.265992 28145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:40559 every 8 connection(s)
I20260926 13:47:26.266923 28016 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:47:26.270501 28148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:26.268938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.66:60105)
0926 13:47:26.269388 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:26.269802 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":100,"threads_started":1}
W20260926 13:47:26.270982 28146 heartbeater.cc:644] Failed to heartbeat to 127.12.170.126:35935 (0 consecutive failures): Network error: Failed to ping master at 127.12.170.126:35935: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
I20260926 13:47:26.271512 28148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:26.271314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.66:46279)
0926 13:47:26.271407 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:26.271453 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 13:47:26.272071 28148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:26.271897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.66:35611)
0926 13:47:26.271986 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:26.272029 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 13:47:26.272208 28146 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 13:47:26.274336 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 28016
I20260926 13:47:26.274672 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:36135
--local_ip_for_outbound_sockets=127.12.170.67
--tserver_master_addrs=127.12.170.126:35935
--webserver_port=41331
--webserver_interface=127.12.170.67
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:47:26.380607 28149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:26.380849 28149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:26.380892 28149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:26.384836 28149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:26.384892 28149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:26.384958 28149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:47:26.390113 28149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:36135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=41331
--tserver_master_addrs=127.12.170.126:35935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:26.391126 28149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:26.392153 28149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:47:26.401594 28154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:26.401597 28155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:26.401713 28149 server_base.cc:1034] running on GCE node
W20260926 13:47:26.401613 28157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:26.402192 28149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:26.402778 28149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:26.403919 28149 hybrid_clock.cc:648] HybridClock initialized: now 1790430446403902 us; error 28 us; skew 500 ppm
Sep 26 13:47:26 dist-test-slave-hc1k krb5kdc[26701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430446, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:26.406458 28149 init.cc:377] Logged in from keytab as kudu/127.12.170.67@KRBTEST.COM (short username kudu)
I20260926 13:47:26.407289 28149 webserver.cc:460] Webserver started at http://127.12.170.67:41331/ using document root <none> and password file <none>
I20260926 13:47:26.407867 28149 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:26.407935 28149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:47:26.411852 28149 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 13:47:26.413748 28164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:26.414896 28149 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:47:26.414994 28149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "7466e77c905546d8a1be7fadd7a5817b"
format_stamp: "Formatted at 2026-09-26 13:47:24 on dist-test-slave-hc1k"
server_key: "7545e64d9744de102e6566d9b29260b8"
server_key_iv: "c5abd7297a1f759f7d65dc25df72f6e3"
server_key_version: "encryptionkey@0"
I20260926 13:47:26.415534 28149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:26.428342 28149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:26.431140 28149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:26.431294 28149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:26.431839 28149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:47:26.432579 28149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:47:26.432623 28149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:26.432672 28149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:47:26.432698 28149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:26.440510 28149 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:36135
I20260926 13:47:26.440536 28277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:36135 every 8 connection(s)
I20260926 13:47:26.441419 28149 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:47:26.444828 28280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:26.443390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.67:42845)
0926 13:47:26.444039 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:26.444150 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":111,"threads_started":1}
W20260926 13:47:26.445218 28278 heartbeater.cc:644] Failed to heartbeat to 127.12.170.126:35935 (0 consecutive failures): Network error: Failed to ping master at 127.12.170.126:35935: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
I20260926 13:47:26.445724 28280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:26.445540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.67:44043)
0926 13:47:26.445631 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:26.445674 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 13:47:26.446223 28280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:26.446052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.67:42757)
0926 13:47:26.446142 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:26.446178 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 13:47:26.446352 28278 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 13:47:26.450541 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 28149
I20260926 13:47:27.127864 28282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:27.127409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.65:38031)
0926 13:47:27.127698 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:27.127758 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":95,"threads_started":1}
I20260926 13:47:27.273056 28283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:27.272701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.66:58555)
0926 13:47:27.272910 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:27.272959 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":66,"threads_started":1}
I20260926 13:47:27.447194 28284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:27.446837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.67:40961)
0926 13:47:27.447044 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:27.447098 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":64,"threads_started":1}
I20260926 13:47:27.824038 28285 raft_consensus.cc:482] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:47:27.824226 28285 raft_consensus.cc:504] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } } peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } }
I20260926 13:47:27.825349 28285 leader_election.cc:288] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7466e77c905546d8a1be7fadd7a5817b (127.12.170.67:36135), 3aee7b81806b4d9ebaff29a885d69ca9 (127.12.170.66:40559)
Sep 26 13:47:27 dist-test-slave-hc1k krb5kdc[26701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430446, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for kudu/127.12.170.66@KRBTEST.COM
Sep 26 13:47:27 dist-test-slave-hc1k krb5kdc[26701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430446, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for kudu/127.12.170.67@KRBTEST.COM
I20260926 13:47:27.838835 28287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:27.825593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:40559 (local address 127.12.170.65:48795)
0926 13:47:27.825933 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:27.825947 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:27.826040 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:27.828390 (+ 2350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:27.828392 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:27.828549 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:27.828885 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:27.828893 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:27.830189 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:27.830191 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:27.831222 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:27.831231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:27.831936 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:27.831939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:27.832120 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:27.832698 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:27.832716 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:27.834230 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:27.836512 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:27.836517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:27.836528 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:27.836764 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:27.837139 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:27.837142 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:27.837143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:27.837274 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:27.837943 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:27.837950 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:27.838161 (+ 211us) client_negotiation.cc:769] Sending connection context
0926 13:47:27.838325 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 13:47:27.838626 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":160,"thread_start_us":75,"threads_started":1}
I20260926 13:47:27.838858 28288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:27.825762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:48795 (local address 127.12.170.66:40559)
0926 13:47:27.826167 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:27.826181 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:27.826221 (+ 40us) server_negotiation.cc:407] Connection header received
0926 13:47:27.827087 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:27.827094 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:27.827546 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:27.827910 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:27.829032 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:27.830063 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:27.831331 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:27.831824 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:27.832003 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:27.834443 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:27.834448 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:27.834459 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:27.834489 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:27.836401 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:27.836878 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:27.836884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:27.836889 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:27.837033 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:27.837364 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:27.837367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:27.837369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:27.837826 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:27.837939 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:27.838452 (+ 513us) server_negotiation.cc:299] Negotiation successful
0926 13:47:27.838701 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":58,"threads_started":1}
I20260926 13:47:27.839071 28289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:27.825732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:34235 (local address 127.12.170.67:36135)
0926 13:47:27.826153 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:27.826172 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:27.826208 (+ 36us) server_negotiation.cc:407] Connection header received
0926 13:47:27.827084 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:27.827094 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:27.827546 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:27.827910 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:27.829028 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:27.830034 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:27.831327 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:27.831825 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:27.831974 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:27.834708 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:27.834714 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:27.834730 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:27.834765 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:27.836401 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:27.837429 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:27.837436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:27.837441 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:27.837595 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:27.837847 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:27.837850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:27.837852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:27.838217 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:27.838324 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:27.838631 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 13:47:27.838915 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":114,"threads_started":1}
I20260926 13:47:27.838878 28286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:27.825576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:36135 (local address 127.12.170.65:34235)
0926 13:47:27.825882 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:27.825896 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:27.826006 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:27.827961 (+ 1955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:27.827966 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:27.828363 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:27.828866 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:27.828881 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:27.830162 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:27.830174 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:27.831216 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:27.831226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:27.831936 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:27.831940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:27.832117 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:27.832754 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:27.832766 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:27.834519 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:27.836875 (+ 2356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:27.836877 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:27.836879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:27.837034 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:27.837687 (+ 653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:27.837690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:27.837692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:27.837768 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:27.838326 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:27.838329 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:27.838417 (+ 88us) client_negotiation.cc:769] Sending connection context
0926 13:47:27.838498 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 13:47:27.838710 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":154,"thread_start_us":74,"threads_started":1}
W20260926 13:47:27.841977 27901 leader_election.cc:341] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7466e77c905546d8a1be7fadd7a5817b (127.12.170.67:36135): Not found: Tablet not found: 895e6e19add44fc7884276270cf8417a
W20260926 13:47:27.842609 27899 leader_election.cc:341] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3aee7b81806b4d9ebaff29a885d69ca9 (127.12.170.66:40559): Not found: Tablet not found: 895e6e19add44fc7884276270cf8417a
I20260926 13:47:27.842705 27899 leader_election.cc:302] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 238606c7eb7c42fdb459e36d055dd84b; no voters: 3aee7b81806b4d9ebaff29a885d69ca9, 7466e77c905546d8a1be7fadd7a5817b
I20260926 13:47:27.842927 28285 raft_consensus.cc:2726] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:47:28.128888 28286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:28.128602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.65:40965)
0926 13:47:28.128738 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:28.128800 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 13:47:28.274297 28290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:28.273821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.66:53579)
0926 13:47:28.274127 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:28.274189 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":74,"threads_started":1}
I20260926 13:47:28.448369 28291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:28.447940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.67:49911)
0926 13:47:28.448207 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:28.448274 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":79,"threads_started":1}
I20260926 13:47:29.129951 28292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:29.129575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.65:43023)
0926 13:47:29.129791 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:29.129845 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":69,"threads_started":1}
I20260926 13:47:29.275593 28293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:29.275128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.66:57349)
0926 13:47:29.275411 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:29.275471 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":76,"threads_started":1}
I20260926 13:47:29.449505 28294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:29.449078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.67:43083)
0926 13:47:29.449350 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:29.449413 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":76,"threads_started":1}
I20260926 13:47:29.476675 28295 raft_consensus.cc:482] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:47:29.476802 28295 raft_consensus.cc:504] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } } peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } }
I20260926 13:47:29.477159 28295 leader_election.cc:288] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7466e77c905546d8a1be7fadd7a5817b (127.12.170.67:36135), 3aee7b81806b4d9ebaff29a885d69ca9 (127.12.170.66:40559)
W20260926 13:47:29.477918 27901 leader_election.cc:341] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7466e77c905546d8a1be7fadd7a5817b (127.12.170.67:36135): Not found: Tablet not found: 895e6e19add44fc7884276270cf8417a
W20260926 13:47:29.478063 27899 leader_election.cc:341] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3aee7b81806b4d9ebaff29a885d69ca9 (127.12.170.66:40559): Not found: Tablet not found: 895e6e19add44fc7884276270cf8417a
I20260926 13:47:29.478181 27899 leader_election.cc:302] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 238606c7eb7c42fdb459e36d055dd84b; no voters: 3aee7b81806b4d9ebaff29a885d69ca9, 7466e77c905546d8a1be7fadd7a5817b
I20260926 13:47:29.478328 28295 raft_consensus.cc:2726] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:47:30.131126 28296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:30.130692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.65:44137)
0926 13:47:30.130935 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:30.131000 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":70,"threads_started":1}
I20260926 13:47:30.276872 28297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:30.276434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.66:58947)
0926 13:47:30.276701 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:30.276762 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":70,"threads_started":1}
I20260926 13:47:30.450631 28298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:30.450231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.67:50365)
0926 13:47:30.450474 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:30.450541 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":68,"threads_started":1}
I20260926 13:47:31.132381 28299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:31.131974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.65:35771)
0926 13:47:31.132238 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:31.132288 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":69,"threads_started":1}
I20260926 13:47:31.277953 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:31.277579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.66:36197)
0926 13:47:31.277812 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:31.277865 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":65,"threads_started":1}
I20260926 13:47:31.451813 28301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:31.451346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.67:42801)
0926 13:47:31.451628 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:31.451716 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":86,"threads_started":1}
I20260926 13:47:32.133378 28302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:32.133002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.65:34179)
0926 13:47:32.133228 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:32.133284 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":71,"threads_started":1}
I20260926 13:47:32.279177 28303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:32.278778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.66:41597)
0926 13:47:32.279023 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:32.279084 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":69,"threads_started":1}
I20260926 13:47:32.421509 28304 raft_consensus.cc:482] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:47:32.421631 28304 raft_consensus.cc:504] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } } peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } }
I20260926 13:47:32.422079 28304 leader_election.cc:288] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7466e77c905546d8a1be7fadd7a5817b (127.12.170.67:36135), 3aee7b81806b4d9ebaff29a885d69ca9 (127.12.170.66:40559)
W20260926 13:47:32.422883 27901 leader_election.cc:341] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7466e77c905546d8a1be7fadd7a5817b (127.12.170.67:36135): Not found: Tablet not found: 895e6e19add44fc7884276270cf8417a
W20260926 13:47:32.423092 27899 leader_election.cc:341] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3aee7b81806b4d9ebaff29a885d69ca9 (127.12.170.66:40559): Not found: Tablet not found: 895e6e19add44fc7884276270cf8417a
I20260926 13:47:32.423162 27899 leader_election.cc:302] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 238606c7eb7c42fdb459e36d055dd84b; no voters: 3aee7b81806b4d9ebaff29a885d69ca9, 7466e77c905546d8a1be7fadd7a5817b
I20260926 13:47:32.423295 28304 raft_consensus.cc:2726] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:47:32.452903 28305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:32.452520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.67:42677)
0926 13:47:32.452759 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:32.452813 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":69,"threads_started":1}
I20260926 13:47:33.134469 28306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:33.134092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.65:58783)
0926 13:47:33.134322 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:33.134376 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":73,"threads_started":1}
I20260926 13:47:33.280310 28307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:33.279871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.66:50837)
0926 13:47:33.280160 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:33.280218 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":65,"threads_started":1}
I20260926 13:47:33.454141 28308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:33.453675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.67:41265)
0926 13:47:33.453989 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:33.454047 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":81,"threads_started":1}
I20260926 13:47:34.135703 28309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:34.135225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.65:35449)
0926 13:47:34.135509 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:34.135571 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":79,"threads_started":1}
I20260926 13:47:34.281463 28310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:34.281012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.66:35739)
0926 13:47:34.281301 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:34.281368 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":91,"threads_started":1}
I20260926 13:47:34.455302 28311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:34.454879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.67:35289)
0926 13:47:34.455156 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:34.455212 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":104,"threads_started":1}
I20260926 13:47:34.689555 28312 raft_consensus.cc:482] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:47:34.689695 28312 raft_consensus.cc:504] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } } peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } }
I20260926 13:47:34.690102 28312 leader_election.cc:288] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7466e77c905546d8a1be7fadd7a5817b (127.12.170.67:36135), 3aee7b81806b4d9ebaff29a885d69ca9 (127.12.170.66:40559)
W20260926 13:47:34.690806 27901 leader_election.cc:341] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7466e77c905546d8a1be7fadd7a5817b (127.12.170.67:36135): Not found: Tablet not found: 895e6e19add44fc7884276270cf8417a
W20260926 13:47:34.690907 27899 leader_election.cc:341] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3aee7b81806b4d9ebaff29a885d69ca9 (127.12.170.66:40559): Not found: Tablet not found: 895e6e19add44fc7884276270cf8417a
I20260926 13:47:34.690990 27899 leader_election.cc:302] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 238606c7eb7c42fdb459e36d055dd84b; no voters: 3aee7b81806b4d9ebaff29a885d69ca9, 7466e77c905546d8a1be7fadd7a5817b
I20260926 13:47:34.691120 28312 raft_consensus.cc:2726] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:47:35.136871 28313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:35.136468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.65:34907)
0926 13:47:35.136713 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:35.136773 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
I20260926 13:47:35.282591 28314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:35.282194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.66:42375)
0926 13:47:35.282447 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:35.282500 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
I20260926 13:47:35.456393 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:35.455982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.67:47229)
0926 13:47:35.456257 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:35.456308 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":100,"threads_started":1}
I20260926 13:47:36.138217 28316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:36.137784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.65:36147)
0926 13:47:36.138036 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:36.138122 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":84,"threads_started":1}
I20260926 13:47:36.283775 28317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:36.283337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.66:47859)
0926 13:47:36.283587 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:36.283644 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":72,"threads_started":1}
I20260926 13:47:36.457692 28318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:36.457195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.67:32987)
0926 13:47:36.457497 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:36.457577 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":85,"threads_started":1}
I20260926 13:47:37.139405 28319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:37.138958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.65:33495)
0926 13:47:37.139256 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:37.139309 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":126,"threads_started":1}
I20260926 13:47:37.284924 28320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:37.284522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.66:58183)
0926 13:47:37.284780 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:37.284834 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":80,"threads_started":1}
I20260926 13:47:37.458884 28321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:37.458460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.67:42767)
0926 13:47:37.458726 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:37.458782 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":86,"threads_started":1}
I20260926 13:47:38.140617 28322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:38.140144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.65:56181)
0926 13:47:38.140453 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:38.140514 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":82,"threads_started":1}
I20260926 13:47:38.285983 28323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:38.285606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.66:37771)
0926 13:47:38.285830 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:38.285886 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":70,"threads_started":1}
I20260926 13:47:38.459997 28324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:38.459581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.67:36629)
0926 13:47:38.459853 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:38.459906 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":95,"threads_started":1}
I20260926 13:47:39.141808 28325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:39.141395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.65:59423)
0926 13:47:39.141659 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:39.141717 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":82,"threads_started":1}
I20260926 13:47:39.287200 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:39.286765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.66:40251)
0926 13:47:39.287025 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:39.287105 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":73,"threads_started":1}
I20260926 13:47:39.461146 28327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:39.460727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.67:56483)
0926 13:47:39.460985 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:39.461052 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":85,"threads_started":1}
I20260926 13:47:39.606570 28328 raft_consensus.cc:482] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:47:39.606700 28328 raft_consensus.cc:504] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7466e77c905546d8a1be7fadd7a5817b" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 36135 } } peers { permanent_uuid: "3aee7b81806b4d9ebaff29a885d69ca9" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 40559 } } peers { permanent_uuid: "238606c7eb7c42fdb459e36d055dd84b" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39299 } }
I20260926 13:47:39.607065 28328 leader_election.cc:288] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7466e77c905546d8a1be7fadd7a5817b (127.12.170.67:36135), 3aee7b81806b4d9ebaff29a885d69ca9 (127.12.170.66:40559)
W20260926 13:47:39.607800 27899 leader_election.cc:341] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3aee7b81806b4d9ebaff29a885d69ca9 (127.12.170.66:40559): Not found: Tablet not found: 895e6e19add44fc7884276270cf8417a
W20260926 13:47:39.607903 27901 leader_election.cc:341] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7466e77c905546d8a1be7fadd7a5817b (127.12.170.67:36135): Not found: Tablet not found: 895e6e19add44fc7884276270cf8417a
I20260926 13:47:39.607988 27901 leader_election.cc:302] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 238606c7eb7c42fdb459e36d055dd84b; no voters: 3aee7b81806b4d9ebaff29a885d69ca9, 7466e77c905546d8a1be7fadd7a5817b
I20260926 13:47:39.608116 28328 raft_consensus.cc:2726] T 895e6e19add44fc7884276270cf8417a P 238606c7eb7c42fdb459e36d055dd84b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:47:40.142988 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:40.142543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.65:43979)
0926 13:47:40.142833 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:40.142893 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":105,"threads_started":1}
I20260926 13:47:40.288363 28330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:40.287961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.66:42163)
0926 13:47:40.288203 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:40.288265 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
I20260926 13:47:40.462435 28331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:40.461933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.67:35007)
0926 13:47:40.462257 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:40.462333 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":125,"threads_started":1}
I20260926 13:47:41.144279 28332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:41.143874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.65:46105)
0926 13:47:41.144130 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:41.144187 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":77,"threads_started":1}
I20260926 13:47:41.289678 28333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:41.289271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:35935 (local address 127.12.170.66:59333)
0926 13:47:41.289526 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:41.289583 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:35935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":72,"threads_started":1}
I20260926 13:47:41.451378 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 27848
I20260926 13:47:41.456338 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 28016
I20260926 13:47:41.460716 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 28149
2026-09-26T13:47:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19308 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:47:41 dist-test-slave-hc1k krb5kdc[28336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:47:41 dist-test-slave-hc1k krb5kdc[28336](info): set up 2 sockets
Sep 26 13:47:41 dist-test-slave-hc1k krb5kdc[28336](info): commencing operation
krb5kdc: starting...
W20260926 13:47:42.423770 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:47:42 dist-test-slave-hc1k krb5kdc[28336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:47:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:47:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:47:42.558485 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:34267
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:34267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:47:42.664227 28349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:42.664471 28349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:42.664523 28349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:42.668427 28349 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 13:47:42.668478 28349 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 13:47:42.668499 28349 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 13:47:42.668515 28349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:47:42.668540 28349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:42.668571 28349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:42.668601 28349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:47:42.668623 28349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:47:42.673703 28349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:34267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:34267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:42.674661 28349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:42.675632 28349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:47:42.683537 28354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:42.683631 28349 server_base.cc:1034] running on GCE node
W20260926 13:47:42.683519 28355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:42.683519 28357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:42.684399 28349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:42.685389 28349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:42.686555 28349 hybrid_clock.cc:648] HybridClock initialized: now 1790430462686541 us; error 27 us; skew 500 ppm
Sep 26 13:47:42 dist-test-slave-hc1k krb5kdc[28336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430462, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:42.689262 28349 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:47:42.690090 28349 webserver.cc:460] Webserver started at http://127.12.170.126:38535/ using document root <none> and password file <none>
I20260926 13:47:42.690611 28349 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:42.690673 28349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:47:42.690850 28349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:47:42.692812 28349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "6d408b64f6fa430295bfcf3ff8e058cb"
format_stamp: "Formatted at 2026-09-26 13:47:42 on dist-test-slave-hc1k"
server_key: "0d55963306c5444d8aa0d1a4d17bbca9"
server_key_iv: "d1db2af6bdc006f9d1d991b5b09f7180"
server_key_version: "encryptionkey@0"
I20260926 13:47:42.693357 28349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "6d408b64f6fa430295bfcf3ff8e058cb"
format_stamp: "Formatted at 2026-09-26 13:47:42 on dist-test-slave-hc1k"
server_key: "0d55963306c5444d8aa0d1a4d17bbca9"
server_key_iv: "d1db2af6bdc006f9d1d991b5b09f7180"
server_key_version: "encryptionkey@0"
I20260926 13:47:42.696762 28349 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:47:42.698961 28364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:42.700145 28349 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:47:42.700248 28349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "6d408b64f6fa430295bfcf3ff8e058cb"
format_stamp: "Formatted at 2026-09-26 13:47:42 on dist-test-slave-hc1k"
server_key: "0d55963306c5444d8aa0d1a4d17bbca9"
server_key_iv: "d1db2af6bdc006f9d1d991b5b09f7180"
server_key_version: "encryptionkey@0"
I20260926 13:47:42.700330 28349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:42.712383 28349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:42.715184 28349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:42.715343 28349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:42.721524 28349 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:34267
I20260926 13:47:42.721783 28416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:34267 every 8 connection(s)
I20260926 13:47:42.722406 28349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:47:42.723572 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 28349
I20260926 13:47:42.723704 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:47:42.723963 12969 external_mini_cluster.cc:1428] Setting key 277fbc192cef6e67a08afb8efb519683
I20260926 13:47:42.725335 28417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:47:42.731498 28417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb: Bootstrap starting.
Sep 26 13:47:42 dist-test-slave-hc1k krb5kdc[28336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:47:42.733939 28417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb: Neither blocks nor log segments found. Creating new log.
I20260926 13:47:42.734642 28417 log.cc:824] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb: Log is configured to *not* fsync() on all Append() calls
I20260926 13:47:42.736671 28417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb: No bootstrap required, opened a new log
I20260926 13:47:42.737564 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:42.725324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53894 (local address 127.12.170.126:34267)
0926 13:47:42.725696 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:42.725713 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:42.725747 (+ 34us) server_negotiation.cc:407] Connection header received
0926 13:47:42.726350 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:42.726365 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:42.726686 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:42.726942 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:42.727927 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:42.728904 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:42.730074 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:42.730511 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:42.730632 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:42.733422 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:42.733427 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:42.733439 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:42.733463 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:42.735058 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:42.735489 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:42.735494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:42.735498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:42.735604 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:42.735898 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:42.735900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:42.735901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:42.736247 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:42.736376 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:42.736679 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 13:47:42.736917 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":112,"threads_started":1}
I20260926 13:47:42.740135 28417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d408b64f6fa430295bfcf3ff8e058cb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34267 } }
I20260926 13:47:42.740339 28417 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:47:42.740392 28417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d408b64f6fa430295bfcf3ff8e058cb, State: Initialized, Role: FOLLOWER
I20260926 13:47:42.740825 28417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d408b64f6fa430295bfcf3ff8e058cb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34267 } }
I20260926 13:47:42.740975 28417 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:47:42.741046 28417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:47:42.741146 28417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:47:42.741974 28417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d408b64f6fa430295bfcf3ff8e058cb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34267 } }
I20260926 13:47:42.742281 28417 leader_election.cc:302] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d408b64f6fa430295bfcf3ff8e058cb; no voters:
I20260926 13:47:42.742554 28417 leader_election.cc:288] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:47:42.742637 28422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:47:42.742851 28422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [term 1 LEADER]: Becoming Leader. State: Replica: 6d408b64f6fa430295bfcf3ff8e058cb, State: Running, Role: LEADER
I20260926 13:47:42.743153 28422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d408b64f6fa430295bfcf3ff8e058cb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34267 } }
I20260926 13:47:42.743635 28417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:47:42.744742 28423 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d408b64f6fa430295bfcf3ff8e058cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d408b64f6fa430295bfcf3ff8e058cb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34267 } } }
I20260926 13:47:42.744788 28424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d408b64f6fa430295bfcf3ff8e058cb. Latest consensus state: current_term: 1 leader_uuid: "6d408b64f6fa430295bfcf3ff8e058cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d408b64f6fa430295bfcf3ff8e058cb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34267 } } }
I20260926 13:47:42.744937 28423 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [sys.catalog]: This master's current role is: LEADER
I20260926 13:47:42.744966 28424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [sys.catalog]: This master's current role is: LEADER
I20260926 13:47:42.745769 28431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:47:42.748312 28431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:47:42.752787 28431 catalog_manager.cc:1347] Generated new cluster ID: ca6bcd6339d44dd5bfea418a807307e3
I20260926 13:47:42.752848 28431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:47:42.764286 28431 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:47:42.765071 28431 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:47:42.774471 28431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb: Generated new TSK 0
I20260926 13:47:42.775190 28431 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.170.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
I20260926 13:47:42.798462 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:0
--local_ip_for_outbound_sockets=127.12.170.65
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:34267
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:47:42.901379 28443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:42.901600 28443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:42.901639 28443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:42.905617 28443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:42.905664 28443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:42.905717 28443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:47:42.910609 28443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:34267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:42.911506 28443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:42.912449 28443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:47:42.921319 28451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:42.921367 28448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:42.921330 28449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:42.921418 28443 server_base.cc:1034] running on GCE node
I20260926 13:47:42.921897 28443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:42.922483 28443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:42.923631 28443 hybrid_clock.cc:648] HybridClock initialized: now 1790430462923617 us; error 31 us; skew 500 ppm
Sep 26 13:47:42 dist-test-slave-hc1k krb5kdc[28336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430462, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:42.926259 28443 init.cc:377] Logged in from keytab as kudu/127.12.170.65@KRBTEST.COM (short username kudu)
I20260926 13:47:42.927011 28443 webserver.cc:460] Webserver started at http://127.12.170.65:41031/ using document root <none> and password file <none>
I20260926 13:47:42.927511 28443 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:42.927570 28443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:47:42.927783 28443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:47:42.929641 28443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/data/instance:
uuid: "45aabb04e4564deda00644b409c8a912"
format_stamp: "Formatted at 2026-09-26 13:47:42 on dist-test-slave-hc1k"
server_key: "9b887b3e6c8a71e9bd4fb92dbdaca565"
server_key_iv: "f59e86eb5f33d71621bd86a0949e826b"
server_key_version: "encryptionkey@0"
I20260926 13:47:42.930154 28443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance:
uuid: "45aabb04e4564deda00644b409c8a912"
format_stamp: "Formatted at 2026-09-26 13:47:42 on dist-test-slave-hc1k"
server_key: "9b887b3e6c8a71e9bd4fb92dbdaca565"
server_key_iv: "f59e86eb5f33d71621bd86a0949e826b"
server_key_version: "encryptionkey@0"
I20260926 13:47:42.933615 28443 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:47:42.935829 28458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:42.936995 28443 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 13:47:42.937129 28443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "45aabb04e4564deda00644b409c8a912"
format_stamp: "Formatted at 2026-09-26 13:47:42 on dist-test-slave-hc1k"
server_key: "9b887b3e6c8a71e9bd4fb92dbdaca565"
server_key_iv: "f59e86eb5f33d71621bd86a0949e826b"
server_key_version: "encryptionkey@0"
I20260926 13:47:42.937222 28443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:42.948406 28443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:42.951085 28443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:42.951251 28443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:42.951759 28443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:47:42.952471 28443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:47:42.952514 28443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:42.952561 28443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:47:42.952591 28443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:42.960644 28443 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:41083
I20260926 13:47:42.960678 28571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:41083 every 8 connection(s)
I20260926 13:47:42.961398 28443 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:47:42.963410 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 28443
I20260926 13:47:42.963536 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance
I20260926 13:47:42.963878 12969 external_mini_cluster.cc:1428] Setting key b1a2511446a05bc39765930797868f4f
Sep 26 13:47:42 dist-test-slave-hc1k krb5kdc[28336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430462, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.170.66@KRBTEST.COM; defaulting to no policy
I20260926 13:47:42.975006 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:42.963067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:47301 (local address 127.12.170.126:34267)
0926 13:47:42.963221 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:42.963224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:42.963992 (+ 768us) server_negotiation.cc:407] Connection header received
0926 13:47:42.965391 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:42.965395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:42.965455 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:42.965537 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:42.966763 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:42.967407 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:42.968991 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:42.969235 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:42.969312 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:42.971994 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:42.971997 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:42.971999 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:42.972022 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:42.973308 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:42.973761 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:42.973763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:42.973765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:42.973831 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:42.974126 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:42.974129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:42.974130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:42.974288 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:42.974357 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:42.974766 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 13:47:42.974869 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.66@KRBTEST.COM" created.
I20260926 13:47:42.975733 28574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:42.963377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.65:47301)
0926 13:47:42.963854 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:42.963895 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:42.965161 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:42.965710 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:42.965717 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:42.966054 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:42.966598 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:42.966618 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:42.967543 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:42.967547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:42.968865 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:42.968877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:42.969384 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:42.969388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:42.969593 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:42.970126 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:42.970148 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:42.971834 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:42.973420 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:42.973424 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:42.973433 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:42.973678 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:42.973892 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:42.973894 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:42.973895 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:42.974029 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:42.974361 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:42.974365 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:42.974577 (+ 212us) client_negotiation.cc:769] Sending connection context
0926 13:47:42.974713 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 13:47:42.974957 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":121,"threads_started":1}
I20260926 13:47:42.976897 28572 heartbeater.cc:347] Connected to a master server at 127.12.170.126:34267
I20260926 13:47:42.977099 28572 heartbeater.cc:464] Registering TS with master...
I20260926 13:47:42.977514 28572 heartbeater.cc:511] Master 127.12.170.126:34267 requested a full tablet report, sending...
I20260926 13:47:42.978900 28381 ts_manager.cc:194] Registered new tserver with Master: 45aabb04e4564deda00644b409c8a912 (127.12.170.65:41083)
I20260926 13:47:42.980007 28381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.65@KRBTEST.COM'} at 127.12.170.65:47301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
I20260926 13:47:42.990123 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:0
--local_ip_for_outbound_sockets=127.12.170.66
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:34267
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:47:43.092177 28577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:43.092384 28577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:43.092422 28577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:43.096182 28577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:43.096225 28577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:43.096276 28577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:47:43.101096 28577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:34267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:43.101886 28577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:43.102716 28577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:47:43.111405 28585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:43.111443 28582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:43.111443 28583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:43.111508 28577 server_base.cc:1034] running on GCE node
I20260926 13:47:43.112025 28577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:43.112560 28577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:43.113715 28577 hybrid_clock.cc:648] HybridClock initialized: now 1790430463113698 us; error 25 us; skew 500 ppm
Sep 26 13:47:43 dist-test-slave-hc1k krb5kdc[28336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430463, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:43.116253 28577 init.cc:377] Logged in from keytab as kudu/127.12.170.66@KRBTEST.COM (short username kudu)
I20260926 13:47:43.117002 28577 webserver.cc:460] Webserver started at http://127.12.170.66:37053/ using document root <none> and password file <none>
I20260926 13:47:43.117473 28577 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:43.117527 28577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:47:43.117736 28577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:47:43.119629 28577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/data/instance:
uuid: "df30cd17e99f4882b7db22578c23543f"
format_stamp: "Formatted at 2026-09-26 13:47:43 on dist-test-slave-hc1k"
server_key: "391c83058789f24e3e67d28c92174810"
server_key_iv: "a551dfd45eec0013e972eca479ed0a6d"
server_key_version: "encryptionkey@0"
I20260926 13:47:43.120138 28577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance:
uuid: "df30cd17e99f4882b7db22578c23543f"
format_stamp: "Formatted at 2026-09-26 13:47:43 on dist-test-slave-hc1k"
server_key: "391c83058789f24e3e67d28c92174810"
server_key_iv: "a551dfd45eec0013e972eca479ed0a6d"
server_key_version: "encryptionkey@0"
I20260926 13:47:43.123420 28577 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:47:43.125645 28592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:43.126750 28577 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:47:43.126837 28577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "df30cd17e99f4882b7db22578c23543f"
format_stamp: "Formatted at 2026-09-26 13:47:43 on dist-test-slave-hc1k"
server_key: "391c83058789f24e3e67d28c92174810"
server_key_iv: "a551dfd45eec0013e972eca479ed0a6d"
server_key_version: "encryptionkey@0"
I20260926 13:47:43.126931 28577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:43.161073 28577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:43.164005 28577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:43.164170 28577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:43.164722 28577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:47:43.165480 28577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:47:43.165524 28577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:43.165585 28577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:47:43.165614 28577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:43.173655 28577 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:34943
I20260926 13:47:43.173686 28705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:34943 every 8 connection(s)
I20260926 13:47:43.174431 28577 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:47:43.175365 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 28577
I20260926 13:47:43.175482 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance
I20260926 13:47:43.175799 12969 external_mini_cluster.cc:1428] Setting key 1336a92fada3d864144df8a6b83d623a
Sep 26 13:47:43 dist-test-slave-hc1k krb5kdc[28336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430463, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.170.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.67@KRBTEST.COM" created.
I20260926 13:47:43.187963 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:43.176233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:57633 (local address 127.12.170.126:34267)
0926 13:47:43.176359 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:43.176362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:43.177177 (+ 815us) server_negotiation.cc:407] Connection header received
0926 13:47:43.178426 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:43.178429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:43.178481 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:43.178558 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:43.179700 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.180218 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.181811 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.182027 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.182111 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:43.184825 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:43.184828 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:43.184830 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:43.184853 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:43.186147 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:43.186583 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:43.186585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:43.186587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:43.186653 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:43.186914 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:43.186916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:43.186917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:43.187065 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:43.187154 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:43.187688 (+ 534us) server_negotiation.cc:299] Negotiation successful
0926 13:47:43.187808 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:47:43.188730 28708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:43.176618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.66:57633)
0926 13:47:43.177043 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:43.177091 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:43.178224 (+ 1133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:43.178731 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:43.178737 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:43.179058 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:43.179515 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:43.179533 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.180335 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.180339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:43.181698 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:43.181710 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.182170 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.182174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:43.182366 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:43.182881 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:43.182900 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:43.184637 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:43.186251 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:43.186256 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:43.186266 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:43.186485 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:43.186723 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:43.186725 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:43.186727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:43.186847 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:43.187193 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:43.187207 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:43.187437 (+ 230us) client_negotiation.cc:769] Sending connection context
0926 13:47:43.187600 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 13:47:43.187919 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":94,"threads_started":1}
I20260926 13:47:43.189881 28706 heartbeater.cc:347] Connected to a master server at 127.12.170.126:34267
I20260926 13:47:43.190088 28706 heartbeater.cc:464] Registering TS with master...
I20260926 13:47:43.190506 28706 heartbeater.cc:511] Master 127.12.170.126:34267 requested a full tablet report, sending...
I20260926 13:47:43.191498 28381 ts_manager.cc:194] Registered new tserver with Master: df30cd17e99f4882b7db22578c23543f (127.12.170.66:34943)
I20260926 13:47:43.192063 28381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.66@KRBTEST.COM'} at 127.12.170.66:57633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
I20260926 13:47:43.202790 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:0
--local_ip_for_outbound_sockets=127.12.170.67
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:34267
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:47:43.310020 28711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:47:43.310263 28711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:47:43.310447 28711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:47:43.314471 28711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:47:43.314527 28711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:47:43.314586 28711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:47:43.319782 28711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:34267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:47:43.320806 28711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:47:43.321807 28711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:47:43.331202 28717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:47:43.331234 28716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:43.331362 28711 server_base.cc:1034] running on GCE node
W20260926 13:47:43.331543 28719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:47:43.331914 28711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:47:43.332479 28711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:47:43.333637 28711 hybrid_clock.cc:648] HybridClock initialized: now 1790430463333618 us; error 30 us; skew 500 ppm
Sep 26 13:47:43 dist-test-slave-hc1k krb5kdc[28336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430463, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:47:43.336397 28711 init.cc:377] Logged in from keytab as kudu/127.12.170.67@KRBTEST.COM (short username kudu)
I20260926 13:47:43.337181 28711 webserver.cc:460] Webserver started at http://127.12.170.67:40893/ using document root <none> and password file <none>
I20260926 13:47:43.337709 28711 fs_manager.cc:359] Metadata directory not provided
I20260926 13:47:43.337770 28711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:47:43.337946 28711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:47:43.339896 28711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/data/instance:
uuid: "59ec763eef0c4536a092003a70e34f06"
format_stamp: "Formatted at 2026-09-26 13:47:43 on dist-test-slave-hc1k"
server_key: "f15f1e5f70913431aaab7aa764669a84"
server_key_iv: "77dec03e1497067074d8041b5eda4737"
server_key_version: "encryptionkey@0"
I20260926 13:47:43.340438 28711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance:
uuid: "59ec763eef0c4536a092003a70e34f06"
format_stamp: "Formatted at 2026-09-26 13:47:43 on dist-test-slave-hc1k"
server_key: "f15f1e5f70913431aaab7aa764669a84"
server_key_iv: "77dec03e1497067074d8041b5eda4737"
server_key_version: "encryptionkey@0"
I20260926 13:47:43.343914 28711 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 13:47:43.346167 28726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:43.347295 28711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:47:43.347394 28711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "59ec763eef0c4536a092003a70e34f06"
format_stamp: "Formatted at 2026-09-26 13:47:43 on dist-test-slave-hc1k"
server_key: "f15f1e5f70913431aaab7aa764669a84"
server_key_iv: "77dec03e1497067074d8041b5eda4737"
server_key_version: "encryptionkey@0"
I20260926 13:47:43.347481 28711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:47:43.362924 28711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:47:43.365803 28711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:47:43.365957 28711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:47:43.366463 28711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:47:43.367214 28711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:47:43.367259 28711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:43.367296 28711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:47:43.367309 28711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:47:43.375200 28711 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:42981
I20260926 13:47:43.375232 28839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:42981 every 8 connection(s)
I20260926 13:47:43.376048 28711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:47:43.378408 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 28711
I20260926 13:47:43.378505 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance
I20260926 13:47:43.378777 12969 external_mini_cluster.cc:1428] Setting key db7534755abb1e1b8081508d4e4cb0ae
Sep 26 13:47:43 dist-test-slave-hc1k krb5kdc[28336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430463, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:47:43.388411 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:43.377747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:49253 (local address 127.12.170.126:34267)
0926 13:47:43.377874 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:43.377878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:43.378682 (+ 804us) server_negotiation.cc:407] Connection header received
0926 13:47:43.380120 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:43.380123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:43.380171 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:43.380236 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:43.381491 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.382004 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.383085 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.383353 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.383440 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:43.385637 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:43.385640 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:43.385642 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:43.385665 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:43.386751 (+ 1086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:43.387177 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:43.387180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:43.387181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:43.387264 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:43.387511 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:43.387513 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:43.387514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:43.387727 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:43.387801 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:43.388192 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 13:47:43.388297 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 13:47:43.389123 28842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:43.378058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.67:49253)
0926 13:47:43.378513 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:43.378569 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:43.379916 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:43.380432 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:43.380441 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:43.380914 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:47:43.381373 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:43.381386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.382108 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.382111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:43.382964 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:43.382972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.383474 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.383477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:43.383616 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:43.384140 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:47:43.384154 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:47:43.385501 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:47:43.386845 (+ 1344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:43.386850 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:43.386860 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:43.387080 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:43.387336 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:47:43.387338 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:47:43.387339 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:47:43.387448 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:47:43.387802 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:47:43.387807 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:47:43.388012 (+ 205us) client_negotiation.cc:769] Sending connection context
0926 13:47:43.388153 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 13:47:43.388383 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":107,"threads_started":1}
I20260926 13:47:43.390121 28840 heartbeater.cc:347] Connected to a master server at 127.12.170.126:34267
I20260926 13:47:43.390321 28840 heartbeater.cc:464] Registering TS with master...
I20260926 13:47:43.390749 28840 heartbeater.cc:511] Master 127.12.170.126:34267 requested a full tablet report, sending...
I20260926 13:47:43.391798 28381 ts_manager.cc:194] Registered new tserver with Master: 59ec763eef0c4536a092003a70e34f06 (127.12.170.67:42981)
I20260926 13:47:43.392411 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:47:43.392443 28381 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.67@KRBTEST.COM'} at 127.12.170.67:49253
I20260926 13:47:43.400527 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:43.394424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53906 (local address 127.12.170.126:34267)
0926 13:47:43.394545 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:43.394549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:43.394689 (+ 140us) server_negotiation.cc:407] Connection header received
0926 13:47:43.394770 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:43.394773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:43.394805 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:43.394862 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:47:43.395440 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.395971 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.396731 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.396931 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.397003 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:43.398073 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:47:43.398076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:47:43.398078 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:47:43.398098 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:47:43.399204 (+ 1106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:43.399549 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:43.399551 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:43.399552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:43.399616 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:47:43.399848 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:47:43.399850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:47:43.399851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:47:43.400038 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:47:43.400140 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:43.400315 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 13:47:43.400394 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 13:47:43.403051 28381 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:47:43.406131 28381 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:47:43.419770 28852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:43.414400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50938 (local address 127.12.170.67:42981)
0926 13:47:43.414811 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:43.414824 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:43.414847 (+ 23us) server_negotiation.cc:407] Connection header received
0926 13:47:43.414925 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:43.414931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:43.415153 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:43.415311 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:47:43.415818 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.416766 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.418193 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.419020 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.419121 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:43.419264 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:43.419485 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 13:47:43.419586 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":180,"threads_started":1}
I20260926 13:47:43.419790 28851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:43.414233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:42981 (local address 127.0.0.1:50938)
0926 13:47:43.414586 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:43.414641 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:43.414777 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:43.415494 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:43.415500 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:43.415523 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:47:43.415690 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:43.415698 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.416892 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.416894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:43.418068 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:43.418078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.419143 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.419147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:43.419266 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:43.419321 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 13:47:43.419441 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 13:47:43.419580 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":23,"thread_start_us":86,"threads_started":1}
I20260926 13:47:43.420117 28853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:43.414916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:34943 (local address 127.0.0.1:47968)
0926 13:47:43.415176 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:43.415191 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:43.415282 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:43.416062 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:43.416065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:43.416085 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:47:43.416206 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:43.416215 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.417561 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.417565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:43.418695 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:43.418704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.419776 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.419780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:43.419867 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:43.419893 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 13:47:43.419945 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 13:47:43.420005 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":21,"thread_start_us":91,"threads_started":1}
I20260926 13:47:43.420763 28855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:43.415208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47968 (local address 127.12.170.66:34943)
0926 13:47:43.415547 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:43.415560 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:43.415585 (+ 25us) server_negotiation.cc:407] Connection header received
0926 13:47:43.415689 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:43.415694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:43.415894 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:43.416091 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:47:43.416325 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.417424 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.418828 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.419629 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.420370 (+ 741us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:43.420504 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:43.420547 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 13:47:43.420631 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":126,"threads_started":1}
I20260926 13:47:43.420857 28854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:43.415216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:41083 (local address 127.0.0.1:54334)
0926 13:47:43.416280 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:43.416292 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:43.416378 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:43.416842 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:43.416845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:43.416863 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:47:43.416987 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:43.416994 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.418159 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.418163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:43.419138 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:43.419147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.420488 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.420491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:43.420562 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:43.420580 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 13:47:43.420653 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 13:47:43.420722 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"mutex_wait_us":29,"thread_start_us":88,"threads_started":1}
I20260926 13:47:43.420945 28856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:43.415764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54334 (local address 127.12.170.65:41083)
0926 13:47:43.416141 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:43.416151 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:43.416374 (+ 223us) server_negotiation.cc:407] Connection header received
0926 13:47:43.416480 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:43.416486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:43.416693 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:43.416854 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:47:43.417110 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.417966 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.419287 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.420185 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.420295 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:43.420433 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:43.420706 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 13:47:43.420802 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":110,"threads_started":1}
I20260926 13:47:43.423110 28774 tablet_service.cc:1467] Processing CreateTablet for tablet 0d73a8de60a6462f974e1ed6972212db (DEFAULT_TABLE table=test-table [id=aeb0312c42d94171b81741bf4d797fc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:47:43.424065 28774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d73a8de60a6462f974e1ed6972212db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:47:43.423605 28506 tablet_service.cc:1467] Processing CreateTablet for tablet 0d73a8de60a6462f974e1ed6972212db (DEFAULT_TABLE table=test-table [id=aeb0312c42d94171b81741bf4d797fc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:47:43.424656 28506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d73a8de60a6462f974e1ed6972212db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:47:43.424316 28640 tablet_service.cc:1467] Processing CreateTablet for tablet 0d73a8de60a6462f974e1ed6972212db (DEFAULT_TABLE table=test-table [id=aeb0312c42d94171b81741bf4d797fc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:47:43.425329 28640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d73a8de60a6462f974e1ed6972212db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:47:43.429883 28857 tablet_bootstrap.cc:492] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06: Bootstrap starting.
I20260926 13:47:43.432178 28858 tablet_bootstrap.cc:492] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912: Bootstrap starting.
I20260926 13:47:43.432709 28859 tablet_bootstrap.cc:492] T 0d73a8de60a6462f974e1ed6972212db P df30cd17e99f4882b7db22578c23543f: Bootstrap starting.
I20260926 13:47:43.432837 28857 tablet_bootstrap.cc:654] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06: Neither blocks nor log segments found. Creating new log.
I20260926 13:47:43.433542 28857 log.cc:824] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06: Log is configured to *not* fsync() on all Append() calls
I20260926 13:47:43.434691 28858 tablet_bootstrap.cc:654] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912: Neither blocks nor log segments found. Creating new log.
I20260926 13:47:43.435086 28859 tablet_bootstrap.cc:654] T 0d73a8de60a6462f974e1ed6972212db P df30cd17e99f4882b7db22578c23543f: Neither blocks nor log segments found. Creating new log.
I20260926 13:47:43.435269 28857 tablet_bootstrap.cc:492] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06: No bootstrap required, opened a new log
I20260926 13:47:43.435513 28857 ts_tablet_manager.cc:1397] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 13:47:43.435532 28858 log.cc:824] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912: Log is configured to *not* fsync() on all Append() calls
I20260926 13:47:43.435858 28859 log.cc:824] T 0d73a8de60a6462f974e1ed6972212db P df30cd17e99f4882b7db22578c23543f: Log is configured to *not* fsync() on all Append() calls
I20260926 13:47:43.437566 28859 tablet_bootstrap.cc:492] T 0d73a8de60a6462f974e1ed6972212db P df30cd17e99f4882b7db22578c23543f: No bootstrap required, opened a new log
I20260926 13:47:43.437574 28858 tablet_bootstrap.cc:492] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912: No bootstrap required, opened a new log
I20260926 13:47:43.437744 28859 ts_tablet_manager.cc:1397] T 0d73a8de60a6462f974e1ed6972212db P df30cd17e99f4882b7db22578c23543f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 13:47:43.437744 28858 ts_tablet_manager.cc:1397] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 13:47:43.438766 28857 raft_consensus.cc:348] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:47:43.438990 28857 raft_consensus.cc:374] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:47:43.439059 28857 raft_consensus.cc:729] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59ec763eef0c4536a092003a70e34f06, State: Initialized, Role: FOLLOWER
I20260926 13:47:43.439494 28857 consensus_queue.cc:260] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:47:43.440111 28840 heartbeater.cc:503] Master 127.12.170.126:34267 was elected leader, sending a full tablet report...
I20260926 13:47:43.440429 28857 ts_tablet_manager.cc:1428] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 13:47:43.441636 28859 raft_consensus.cc:348] T 0d73a8de60a6462f974e1ed6972212db P df30cd17e99f4882b7db22578c23543f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:47:43.441897 28859 raft_consensus.cc:374] T 0d73a8de60a6462f974e1ed6972212db P df30cd17e99f4882b7db22578c23543f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:47:43.441977 28859 raft_consensus.cc:729] T 0d73a8de60a6462f974e1ed6972212db P df30cd17e99f4882b7db22578c23543f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df30cd17e99f4882b7db22578c23543f, State: Initialized, Role: FOLLOWER
I20260926 13:47:43.441984 28858 raft_consensus.cc:348] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:47:43.442270 28858 raft_consensus.cc:374] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:47:43.442335 28858 raft_consensus.cc:729] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45aabb04e4564deda00644b409c8a912, State: Initialized, Role: FOLLOWER
I20260926 13:47:43.442421 28859 consensus_queue.cc:260] T 0d73a8de60a6462f974e1ed6972212db P df30cd17e99f4882b7db22578c23543f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:47:43.442729 28858 consensus_queue.cc:260] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:47:43.443202 28706 heartbeater.cc:503] Master 127.12.170.126:34267 was elected leader, sending a full tablet report...
I20260926 13:47:43.443382 28859 ts_tablet_manager.cc:1428] T 0d73a8de60a6462f974e1ed6972212db P df30cd17e99f4882b7db22578c23543f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 13:47:43.443600 28858 ts_tablet_manager.cc:1428] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 13:47:43.443962 28572 heartbeater.cc:503] Master 127.12.170.126:34267 was elected leader, sending a full tablet report...
W20260926 13:47:43.462509 28573 tablet.cc:2369] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:47:43.466966 28863 raft_consensus.cc:482] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:47:43.467177 28863 raft_consensus.cc:504] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:47:43.468364 28863 leader_election.cc:288] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df30cd17e99f4882b7db22578c23543f (127.12.170.66:34943), 45aabb04e4564deda00644b409c8a912 (127.12.170.65:41083)
I20260926 13:47:43.472651 28842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:43.468585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:34943 (local address 127.12.170.67:46569)
0926 13:47:43.468732 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:43.468748 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:43.468845 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:43.469140 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:43.469145 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:43.469163 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:47:43.469302 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:43.469309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.470271 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.470274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:43.471463 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:43.471473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.472283 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.472286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:43.472373 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:43.472393 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 13:47:43.472447 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 13:47:43.472511 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"spinlock_wait_cycles":27520}
I20260926 13:47:43.472695 28855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:43.468690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:46569 (local address 127.12.170.66:34943)
0926 13:47:43.468805 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:43.468809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:43.468825 (+ 16us) server_negotiation.cc:407] Connection header received
0926 13:47:43.468966 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:43.468970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:43.469023 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:43.469107 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:47:43.469434 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.470153 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.471583 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.472170 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.472287 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:43.472340 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:43.472485 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 13:47:43.472572 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 13:47:43.472911 28866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:43.468843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:41083 (local address 127.12.170.67:55365)
0926 13:47:43.469113 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:43.469126 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:47:43.469225 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:47:43.469580 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:47:43.469583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:47:43.469601 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:47:43.469703 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:43.469709 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.470619 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.470622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:43.471755 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:47:43.471763 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.472534 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.472537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:47:43.472639 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:43.472660 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 13:47:43.472729 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 13:47:43.472788 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"spinlock_wait_cycles":2304,"thread_start_us":107,"threads_started":1}
I20260926 13:47:43.473237 28856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:43.468963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:55365 (local address 127.12.170.65:41083)
0926 13:47:43.469103 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:43.469107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:43.469179 (+ 72us) server_negotiation.cc:407] Connection header received
0926 13:47:43.469367 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:43.469371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:43.469429 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:43.469517 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:47:43.469816 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.470515 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.471863 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.472425 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.472534 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:43.472587 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:43.473089 (+ 502us) server_negotiation.cc:299] Negotiation successful
0926 13:47:43.473152 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 13:47:43.473309 28660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d73a8de60a6462f974e1ed6972212db" candidate_uuid: "59ec763eef0c4536a092003a70e34f06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df30cd17e99f4882b7db22578c23543f" is_pre_election: true
I20260926 13:47:43.473626 28660 raft_consensus.cc:2454] T 0d73a8de60a6462f974e1ed6972212db P df30cd17e99f4882b7db22578c23543f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59ec763eef0c4536a092003a70e34f06 in term 0.
I20260926 13:47:43.473793 28526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d73a8de60a6462f974e1ed6972212db" candidate_uuid: "59ec763eef0c4536a092003a70e34f06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45aabb04e4564deda00644b409c8a912" is_pre_election: true
I20260926 13:47:43.474104 28526 raft_consensus.cc:2454] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59ec763eef0c4536a092003a70e34f06 in term 0.
I20260926 13:47:43.474148 28728 leader_election.cc:302] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59ec763eef0c4536a092003a70e34f06, df30cd17e99f4882b7db22578c23543f; no voters:
I20260926 13:47:43.474403 28863 raft_consensus.cc:2781] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:47:43.474502 28863 raft_consensus.cc:482] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:47:43.474565 28863 raft_consensus.cc:3037] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:47:43.475584 28863 raft_consensus.cc:504] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:47:43.475929 28863 leader_election.cc:288] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06 [CANDIDATE]: Term 1 election: Requested vote from peers df30cd17e99f4882b7db22578c23543f (127.12.170.66:34943), 45aabb04e4564deda00644b409c8a912 (127.12.170.65:41083)
I20260926 13:47:43.476308 28526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d73a8de60a6462f974e1ed6972212db" candidate_uuid: "59ec763eef0c4536a092003a70e34f06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45aabb04e4564deda00644b409c8a912"
I20260926 13:47:43.476424 28526 raft_consensus.cc:3037] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:47:43.477393 28526 raft_consensus.cc:2454] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ec763eef0c4536a092003a70e34f06 in term 1.
I20260926 13:47:43.477636 28660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d73a8de60a6462f974e1ed6972212db" candidate_uuid: "59ec763eef0c4536a092003a70e34f06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df30cd17e99f4882b7db22578c23543f"
I20260926 13:47:43.477656 28728 leader_election.cc:302] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45aabb04e4564deda00644b409c8a912, 59ec763eef0c4536a092003a70e34f06; no voters:
I20260926 13:47:43.477737 28660 raft_consensus.cc:3037] T 0d73a8de60a6462f974e1ed6972212db P df30cd17e99f4882b7db22578c23543f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:47:43.477851 28863 raft_consensus.cc:2781] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:47:43.478068 28863 raft_consensus.cc:686] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06 [term 1 LEADER]: Becoming Leader. State: Replica: 59ec763eef0c4536a092003a70e34f06, State: Running, Role: LEADER
I20260926 13:47:43.478420 28863 consensus_queue.cc:237] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:47:43.478942 28660 raft_consensus.cc:2454] T 0d73a8de60a6462f974e1ed6972212db P df30cd17e99f4882b7db22578c23543f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ec763eef0c4536a092003a70e34f06 in term 1.
I20260926 13:47:43.481510 28381 catalog_manager.cc:5495] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59ec763eef0c4536a092003a70e34f06 (127.12.170.67). New cstate: current_term: 1 leader_uuid: "59ec763eef0c4536a092003a70e34f06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } health_report { overall_health: UNKNOWN } } }
I20260926 13:47:43.499454 28852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:43.496207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50940 (local address 127.12.170.67:42981)
0926 13:47:43.496330 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 13:47:43.496335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:47:43.496348 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:47:43.496452 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:47:43.496454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:47:43.496506 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:47:43.496597 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:47:43.496836 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.497324 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.498136 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:47:43.498345 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:47:43.498431 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:47:43.498713 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:47:43.499105 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:47:43.499205 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 13:47:43.499272 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 13:47:43.499326 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 13:47:43.505085 28660 raft_consensus.cc:1261] T 0d73a8de60a6462f974e1ed6972212db P df30cd17e99f4882b7db22578c23543f [term 1 FOLLOWER]: Refusing update from remote peer 59ec763eef0c4536a092003a70e34f06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:47:43.505086 28526 raft_consensus.cc:1261] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912 [term 1 FOLLOWER]: Refusing update from remote peer 59ec763eef0c4536a092003a70e34f06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:47:43.505704 28863 consensus_queue.cc:1035] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:47:43.505954 28867 consensus_queue.cc:1035] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:47:43.512792 28872 mvcc.cc:205] Tried to move back new op lower bound from 7333603178509660160 to 7333603178410704896. Current Snapshot: MvccSnapshot[applied={T|T < 7333603178509660160}]
I20260926 13:47:43.513273 28870 mvcc.cc:205] Tried to move back new op lower bound from 7333603178509660160 to 7333603178410704896. Current Snapshot: MvccSnapshot[applied={T|T < 7333603178509660160}]
I20260926 13:47:43.514931 28871 mvcc.cc:205] Tried to move back new op lower bound from 7333603178509660160 to 7333603178410704896. Current Snapshot: MvccSnapshot[applied={T|T < 7333603178509660160}]
I20260926 13:47:43.520110 28381 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:47:43.520236 28381 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:47:43.522485 28381 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb: Sending DeleteTablet for 3 replicas of tablet 0d73a8de60a6462f974e1ed6972212db
I20260926 13:47:43.523176 28774 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d73a8de60a6462f974e1ed6972212db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:47:43 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:50938
I20260926 13:47:43.523276 28506 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d73a8de60a6462f974e1ed6972212db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:47:43 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:54334
I20260926 13:47:43.523339 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 28443
I20260926 13:47:43.523412 28640 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d73a8de60a6462f974e1ed6972212db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:47:43 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:47968
I20260926 13:47:43.524402 28880 tablet_replica.cc:330] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06: stopping tablet replica
I20260926 13:47:43.524721 28880 raft_consensus.cc:2229] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:47:43.525178 28880 raft_consensus.cc:2258] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:47:43.525573 28881 tablet_replica.cc:330] T 0d73a8de60a6462f974e1ed6972212db P df30cd17e99f4882b7db22578c23543f: stopping tablet replica
I20260926 13:47:43.525902 28881 raft_consensus.cc:2229] T 0d73a8de60a6462f974e1ed6972212db P df30cd17e99f4882b7db22578c23543f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:47:43.526257 28881 raft_consensus.cc:2258] T 0d73a8de60a6462f974e1ed6972212db P df30cd17e99f4882b7db22578c23543f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:47:43.526784 28880 ts_tablet_manager.cc:1905] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:47:43.528272 28881 ts_tablet_manager.cc:1905] T 0d73a8de60a6462f974e1ed6972212db P df30cd17e99f4882b7db22578c23543f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:47:43.529666 28880 ts_tablet_manager.cc:1918] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:47:43.529870 28880 log.cc:1196] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/0d73a8de60a6462f974e1ed6972212db
I20260926 13:47:43.530261 28880 ts_tablet_manager.cc:1939] T 0d73a8de60a6462f974e1ed6972212db P 59ec763eef0c4536a092003a70e34f06: Deleting consensus metadata
W20260926 13:47:43.530781 28366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.170.65:41083 (error 108)
I20260926 13:47:43.530965 28881 ts_tablet_manager.cc:1918] T 0d73a8de60a6462f974e1ed6972212db P df30cd17e99f4882b7db22578c23543f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:47:43.531064 28881 log.cc:1196] T 0d73a8de60a6462f974e1ed6972212db P df30cd17e99f4882b7db22578c23543f: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/wal/wals/0d73a8de60a6462f974e1ed6972212db
I20260926 13:47:43.531373 28881 ts_tablet_manager.cc:1939] T 0d73a8de60a6462f974e1ed6972212db P df30cd17e99f4882b7db22578c23543f: Deleting consensus metadata
I20260926 13:47:43.531611 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 28577
I20260926 13:47:43.531689 28366 catalog_manager.cc:4841] TS 59ec763eef0c4536a092003a70e34f06 (127.12.170.67:42981): tablet 0d73a8de60a6462f974e1ed6972212db (table test-table [id=aeb0312c42d94171b81741bf4d797fc4]) successfully deleted
I20260926 13:47:43.533187 28854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:43.532850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:41083 (local address 127.0.0.1:54342)
0926 13:47:43.533025 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:43.533098 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.65:41083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260926 13:47:43.533358 28366 catalog_manager.cc:4584] TS 45aabb04e4564deda00644b409c8a912 (127.12.170.65:41083): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0d73a8de60a6462f974e1ed6972212db: Network error: Client connection negotiation failed: client connection to 127.12.170.65:41083: connect: Connection refused (error 111)
I20260926 13:47:43.538394 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 28711
I20260926 13:47:43.539224 28854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:47:43.538588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:34943 (local address 127.0.0.1:47972)
0926 13:47:43.539086 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 13:47:43.539144 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.66:34943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420}
I20260926 13:47:43.543397 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 28349
I20260926 13:48:03.548609 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:34267
--webserver_interface=127.12.170.126
--webserver_port=38535
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:34267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:48:03.655174 28883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:03.655412 28883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:03.655467 28883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:03.659422 28883 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 13:48:03.659471 28883 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 13:48:03.659492 28883 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 13:48:03.659508 28883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:48:03.659531 28883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:03.659564 28883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:03.659592 28883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:48:03.659621 28883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:48:03.664714 28883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:34267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:34267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=38535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:03.665696 28883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:03.666648 28883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:03.678697 28890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:03.678730 28889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:03.678699 28883 server_base.cc:1034] running on GCE node
W20260926 13:48:03.678752 28892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:03.679524 28883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:03.680655 28883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:03.681810 28883 hybrid_clock.cc:648] HybridClock initialized: now 1790430483681784 us; error 42 us; skew 500 ppm
Sep 26 13:48:03 dist-test-slave-hc1k krb5kdc[28336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430483, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:03.684655 28883 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:48:03.685542 28883 webserver.cc:460] Webserver started at http://127.12.170.126:38535/ using document root <none> and password file <none>
I20260926 13:48:03.686080 28883 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:03.686153 28883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:03.692962 28883 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260926 13:48:03.695379 28899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:03.696890 28883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 13:48:03.696993 28883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "6d408b64f6fa430295bfcf3ff8e058cb"
format_stamp: "Formatted at 2026-09-26 13:47:42 on dist-test-slave-hc1k"
server_key: "0d55963306c5444d8aa0d1a4d17bbca9"
server_key_iv: "d1db2af6bdc006f9d1d991b5b09f7180"
server_key_version: "encryptionkey@0"
I20260926 13:48:03.697564 28883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:03.711442 28883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:03.714118 28883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:03.714257 28883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:03.720489 28883 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:34267
I20260926 13:48:03.720749 28951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:34267 every 8 connection(s)
I20260926 13:48:03.721479 28883 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:48:03.724601 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 28883
I20260926 13:48:03.724977 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:41083
--local_ip_for_outbound_sockets=127.12.170.65
--tserver_master_addrs=127.12.170.126:34267
--webserver_port=41031
--webserver_interface=127.12.170.65
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:48:03.725710 28952 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:48:03.727675 28952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb: Bootstrap starting.
I20260926 13:48:03.743703 28952 log.cc:824] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:03.752038 28952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:48:03.752377 28952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb: Bootstrap complete.
I20260926 13:48:03.756071 28952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d408b64f6fa430295bfcf3ff8e058cb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34267 } }
I20260926 13:48:03.756457 28952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d408b64f6fa430295bfcf3ff8e058cb, State: Initialized, Role: FOLLOWER
I20260926 13:48:03.756909 28952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d408b64f6fa430295bfcf3ff8e058cb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34267 } }
I20260926 13:48:03.757036 28952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:48:03.757102 28952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:48:03.757205 28952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [term 1 FOLLOWER]: Advancing to term 2
I20260926 13:48:03.758992 28952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d408b64f6fa430295bfcf3ff8e058cb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34267 } }
I20260926 13:48:03.759294 28952 leader_election.cc:302] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d408b64f6fa430295bfcf3ff8e058cb; no voters:
I20260926 13:48:03.759789 28952 leader_election.cc:288] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 13:48:03.760324 28957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [term 2 FOLLOWER]: Leader election won for term 2
I20260926 13:48:03.760784 28957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [term 2 LEADER]: Becoming Leader. State: Replica: 6d408b64f6fa430295bfcf3ff8e058cb, State: Running, Role: LEADER
I20260926 13:48:03.761070 28952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:48:03.761133 28957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d408b64f6fa430295bfcf3ff8e058cb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34267 } }
I20260926 13:48:03.762071 28957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d408b64f6fa430295bfcf3ff8e058cb. Latest consensus state: current_term: 2 leader_uuid: "6d408b64f6fa430295bfcf3ff8e058cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d408b64f6fa430295bfcf3ff8e058cb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34267 } } }
I20260926 13:48:03.762248 28957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:03.762430 28956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6d408b64f6fa430295bfcf3ff8e058cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d408b64f6fa430295bfcf3ff8e058cb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34267 } } }
I20260926 13:48:03.762537 28956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:03.766054 28971 catalog_manager.cc:1259] Loaded cluster ID: ca6bcd6339d44dd5bfea418a807307e3
I20260926 13:48:03.766186 28971 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb: loading cluster ID for follower catalog manager: success
I20260926 13:48:03.768653 28971 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb: acquiring CA information for follower catalog manager: success
I20260926 13:48:03.770090 28971 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:48:03.770210 28965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:48:03.772467 28965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:48:03.772913 28965 catalog_manager.cc:1259] Loaded cluster ID: ca6bcd6339d44dd5bfea418a807307e3
I20260926 13:48:03.772969 28965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:48:03.773582 28965 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:48:03.774034 28965 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb: Loaded TSK: 0
I20260926 13:48:03.793308 28965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb: Generated new TSK 1
I20260926 13:48:03.793542 28965 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 13:48:03.838467 28954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:03.838703 28954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:03.838743 28954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:03.842660 28954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:03.842715 28954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:03.842773 28954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:48:03.847781 28954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:41083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=41031
--tserver_master_addrs=127.12.170.126:34267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:03.848701 28954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:03.849620 28954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:03.858538 28982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:03.858538 28980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:03.858644 28979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:03.858840 28954 server_base.cc:1034] running on GCE node
I20260926 13:48:03.859332 28954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:03.859958 28954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:03.861117 28954 hybrid_clock.cc:648] HybridClock initialized: now 1790430483861099 us; error 34 us; skew 500 ppm
Sep 26 13:48:03 dist-test-slave-hc1k krb5kdc[28336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430483, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:03.864898 28954 init.cc:377] Logged in from keytab as kudu/127.12.170.65@KRBTEST.COM (short username kudu)
I20260926 13:48:03.865957 28954 webserver.cc:460] Webserver started at http://127.12.170.65:41031/ using document root <none> and password file <none>
I20260926 13:48:03.866649 28954 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:03.866753 28954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:03.871588 28954 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:48:03.874100 28989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:03.875485 28954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:48:03.875602 28954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "45aabb04e4564deda00644b409c8a912"
format_stamp: "Formatted at 2026-09-26 13:47:42 on dist-test-slave-hc1k"
server_key: "9b887b3e6c8a71e9bd4fb92dbdaca565"
server_key_iv: "f59e86eb5f33d71621bd86a0949e826b"
server_key_version: "encryptionkey@0"
I20260926 13:48:03.876312 28954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:03.897364 28954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:03.900887 28954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:03.901154 28954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:03.901800 28954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:48:03.903044 28996 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 13:48:03.907364 28954 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 13:48:03.907436 28954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260926 13:48:03.907498 28954 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 13:48:03.909179 28954 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 13:48:03.909220 28954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 13:48:03.909511 28996 tablet_bootstrap.cc:492] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912: Bootstrap starting.
I20260926 13:48:03.918166 28954 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:41083
I20260926 13:48:03.918228 29103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:41083 every 8 connection(s)
I20260926 13:48:03.919097 28954 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:48:03.921314 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 28954
I20260926 13:48:03.921767 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:34943
--local_ip_for_outbound_sockets=127.12.170.66
--tserver_master_addrs=127.12.170.126:34267
--webserver_port=37053
--webserver_interface=127.12.170.66
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:48:03 dist-test-slave-hc1k krb5kdc[28336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430483, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:48:03.930254 28996 log.cc:824] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:03.936403 29106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:03.921533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:47603 (local address 127.12.170.126:34267)
0926 13:48:03.921911 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:03.921930 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:03.922569 (+ 639us) server_negotiation.cc:407] Connection header received
0926 13:48:03.923817 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:03.923834 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:03.924253 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:03.924690 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:03.925700 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:03.926762 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:03.927906 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:03.928364 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:03.928510 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:03.930907 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:03.930913 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:03.930930 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:03.930961 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:03.933002 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:03.933623 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:03.933629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:03.933634 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:03.933785 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:03.934140 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:03.934143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:03.934145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:03.934574 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:03.934713 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:03.935297 (+ 584us) server_negotiation.cc:299] Negotiation successful
0926 13:48:03.935568 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":126,"threads_started":1}
I20260926 13:48:03.936421 29107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:03.921514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.65:47603)
0926 13:48:03.922411 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:03.922456 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:03.923301 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:03.924715 (+ 1414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:03.924722 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:03.925064 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:48:03.925532 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:03.925545 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:03.926909 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:03.926912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:03.927799 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:03.927809 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:03.928498 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:03.928502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:03.928692 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:03.929246 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:48:03.929262 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:48:03.930684 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:48:03.933132 (+ 2448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:03.933139 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:03.933153 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:03.933495 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:03.933879 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:03.933882 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:03.933884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:03.934043 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:03.934698 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:48:03.934704 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:48:03.934976 (+ 272us) client_negotiation.cc:769] Sending connection context
0926 13:48:03.935162 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 13:48:03.935473 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":131,"threads_started":1}
I20260926 13:48:03.938596 28996 tablet_bootstrap.cc:492] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:48:03.939013 28996 tablet_bootstrap.cc:492] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912: Bootstrap complete.
I20260926 13:48:03.939262 29104 heartbeater.cc:347] Connected to a master server at 127.12.170.126:34267
I20260926 13:48:03.939498 29104 heartbeater.cc:464] Registering TS with master...
I20260926 13:48:03.940059 28996 ts_tablet_manager.cc:1397] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912: Time spent bootstrapping tablet: real 0.031s user 0.017s sys 0.012s
I20260926 13:48:03.940100 29104 heartbeater.cc:511] Master 127.12.170.126:34267 requested a full tablet report, sending...
I20260926 13:48:03.942406 28996 raft_consensus.cc:348] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:48:03.942894 28996 raft_consensus.cc:729] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45aabb04e4564deda00644b409c8a912, State: Initialized, Role: FOLLOWER
I20260926 13:48:03.943008 28916 ts_manager.cc:194] Registered new tserver with Master: 45aabb04e4564deda00644b409c8a912 (127.12.170.65:41083)
I20260926 13:48:03.943396 28996 consensus_queue.cc:260] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:48:03.944430 28996 ts_tablet_manager.cc:1428] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20260926 13:48:03.946527 28916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.65@KRBTEST.COM'} at 127.12.170.65:47603
I20260926 13:48:03.949035 29104 heartbeater.cc:503] Master 127.12.170.126:34267 was elected leader, sending a full tablet report...
I20260926 13:48:03.952216 29112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:03.945976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51500 (local address 127.12.170.65:41083)
0926 13:48:03.946320 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:03.946331 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:03.947858 (+ 1527us) server_negotiation.cc:407] Connection header received
0926 13:48:03.947911 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:03.947914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:03.948219 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:03.948380 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:03.948638 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:03.949380 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:03.951081 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:03.951565 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:03.951700 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:03.951812 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:03.952027 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 13:48:03.952118 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":100,"threads_started":1}
I20260926 13:48:03.952241 29111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:03.945883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:41083 (local address 127.0.0.1:51500)
0926 13:48:03.947588 (+ 1705us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:03.947623 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:03.947757 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:03.948384 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:03.948390 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:03.948411 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:03.948524 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:03.948530 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:03.949618 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:03.949622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:03.950946 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:03.950959 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:03.951719 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:03.951723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:03.951836 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:03.951874 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 13:48:03.951981 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 13:48:03.952103 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1566,"thread_start_us":103,"threads_started":1}
I20260926 13:48:03.953332 29038 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d73a8de60a6462f974e1ed6972212db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:47:43 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:51500
I20260926 13:48:03.954084 29113 tablet_replica.cc:330] stopping tablet replica
I20260926 13:48:03.954341 29113 raft_consensus.cc:2229] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:48:03.954553 29113 raft_consensus.cc:2258] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:48:03.955438 29113 ts_tablet_manager.cc:1905] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:48:03.958688 29113 ts_tablet_manager.cc:1918] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:48:03.958760 29113 log.cc:1196] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/wal/wals/0d73a8de60a6462f974e1ed6972212db
I20260926 13:48:03.958979 29113 ts_tablet_manager.cc:1939] T 0d73a8de60a6462f974e1ed6972212db P 45aabb04e4564deda00644b409c8a912: Deleting consensus metadata
I20260926 13:48:03.960840 28901 catalog_manager.cc:4841] TS 45aabb04e4564deda00644b409c8a912 (127.12.170.65:41083): tablet 0d73a8de60a6462f974e1ed6972212db (table test-table [id=aeb0312c42d94171b81741bf4d797fc4]) successfully deleted
W20260926 13:48:04.035429 29108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:04.035702 29108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:04.035763 29108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:04.039690 29108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:04.039744 29108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:04.039822 29108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:48:04.044783 29108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:34943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=37053
--tserver_master_addrs=127.12.170.126:34267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:04.045742 29108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:04.046677 29108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:04.055632 29118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:04.055733 29108 server_base.cc:1034] running on GCE node
W20260926 13:48:04.055817 29119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:04.055996 29121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:04.056358 29108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:04.056936 29108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:04.058084 29108 hybrid_clock.cc:648] HybridClock initialized: now 1790430484058065 us; error 27 us; skew 500 ppm
Sep 26 13:48:04 dist-test-slave-hc1k krb5kdc[28336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430484, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:04.060598 29108 init.cc:377] Logged in from keytab as kudu/127.12.170.66@KRBTEST.COM (short username kudu)
I20260926 13:48:04.061461 29108 webserver.cc:460] Webserver started at http://127.12.170.66:37053/ using document root <none> and password file <none>
I20260926 13:48:04.062014 29108 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:04.062091 29108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:04.065798 29108 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:48:04.067759 29128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:04.068929 29108 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:48:04.069029 29108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "df30cd17e99f4882b7db22578c23543f"
format_stamp: "Formatted at 2026-09-26 13:47:43 on dist-test-slave-hc1k"
server_key: "391c83058789f24e3e67d28c92174810"
server_key_iv: "a551dfd45eec0013e972eca479ed0a6d"
server_key_version: "encryptionkey@0"
I20260926 13:48:04.069532 29108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:04.111115 29108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:04.114078 29108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:04.114256 29108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:04.114836 29108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:48:04.115797 29108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:48:04.115865 29108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:04.115924 29108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:48:04.115950 29108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:04.124534 29108 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:34943
I20260926 13:48:04.124588 29241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:34943 every 8 connection(s)
I20260926 13:48:04.125494 29108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:48:04.130151 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 29108
I20260926 13:48:04.130511 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:42981
--local_ip_for_outbound_sockets=127.12.170.67
--tserver_master_addrs=127.12.170.126:34267
--webserver_port=40893
--webserver_interface=127.12.170.67
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:48:04 dist-test-slave-hc1k krb5kdc[28336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430484, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:48:04.139813 29106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:04.127269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:50669 (local address 127.12.170.126:34267)
0926 13:48:04.127397 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:04.127401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:04.128251 (+ 850us) server_negotiation.cc:407] Connection header received
0926 13:48:04.129467 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:04.129470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:04.129520 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:04.129598 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:04.130783 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.131525 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.133184 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.133437 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.133519 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:04.136293 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:04.136296 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:04.136298 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:04.136322 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:04.137622 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:04.138197 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:04.138200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:04.138201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:04.138281 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:04.138673 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:04.138676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:04.138677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:04.138870 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:04.138954 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:04.139537 (+ 583us) server_negotiation.cc:299] Negotiation successful
0926 13:48:04.139677 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 13:48:04.140911 29244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:04.127643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.66:50669)
0926 13:48:04.128094 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:04.128134 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:04.129246 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:04.129745 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:04.129752 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:04.130088 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:48:04.130627 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:04.130646 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.131691 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.131695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:04.133042 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:04.133054 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.133578 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.133582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:04.133771 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:04.134437 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:48:04.134457 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:48:04.136121 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:48:04.137728 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:04.137734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:04.137750 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:04.138076 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:04.138359 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:04.138362 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:04.138365 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:04.138537 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:04.138971 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:48:04.138978 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:48:04.139298 (+ 320us) client_negotiation.cc:769] Sending connection context
0926 13:48:04.139505 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 13:48:04.139863 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":119,"threads_started":1}
I20260926 13:48:04.142131 29242 heartbeater.cc:347] Connected to a master server at 127.12.170.126:34267
I20260926 13:48:04.142362 29242 heartbeater.cc:464] Registering TS with master...
I20260926 13:48:04.142843 29242 heartbeater.cc:511] Master 127.12.170.126:34267 requested a full tablet report, sending...
I20260926 13:48:04.143877 28916 ts_manager.cc:194] Registered new tserver with Master: df30cd17e99f4882b7db22578c23543f (127.12.170.66:34943)
I20260926 13:48:04.144491 28916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.66@KRBTEST.COM'} at 127.12.170.66:50669
W20260926 13:48:04.241653 29245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:04.241895 29245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:04.241950 29245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:04.245887 29245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:04.245935 29245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:04.246021 29245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:48:04.250972 29245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:42981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=40893
--tserver_master_addrs=127.12.170.126:34267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:04.251943 29245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:04.252977 29245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:04.261807 29250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:04.261945 29245 server_base.cc:1034] running on GCE node
W20260926 13:48:04.261861 29253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:04.262121 29251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:04.262526 29245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:04.263183 29245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:04.264336 29245 hybrid_clock.cc:648] HybridClock initialized: now 1790430484264309 us; error 43 us; skew 500 ppm
Sep 26 13:48:04 dist-test-slave-hc1k krb5kdc[28336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430484, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:04.266901 29245 init.cc:377] Logged in from keytab as kudu/127.12.170.67@KRBTEST.COM (short username kudu)
I20260926 13:48:04.267724 29245 webserver.cc:460] Webserver started at http://127.12.170.67:40893/ using document root <none> and password file <none>
I20260926 13:48:04.268218 29245 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:04.268296 29245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:04.271909 29245 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 13:48:04.273960 29260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:04.275182 29245 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:48:04.275281 29245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "59ec763eef0c4536a092003a70e34f06"
format_stamp: "Formatted at 2026-09-26 13:47:43 on dist-test-slave-hc1k"
server_key: "f15f1e5f70913431aaab7aa764669a84"
server_key_iv: "77dec03e1497067074d8041b5eda4737"
server_key_version: "encryptionkey@0"
I20260926 13:48:04.275910 29245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:04.295615 29245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:04.298604 29245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:04.298777 29245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:04.299326 29245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:48:04.300166 29245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:48:04.300217 29245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:04.300258 29245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:48:04.300285 29245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:04.308399 29245 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:42981
I20260926 13:48:04.308435 29373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:42981 every 8 connection(s)
I20260926 13:48:04.309314 29245 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:48:04.317198 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 29245
Sep 26 13:48:04 dist-test-slave-hc1k krb5kdc[28336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430484, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:48:04.322189 29106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:04.310941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:43503 (local address 127.12.170.126:34267)
0926 13:48:04.311074 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:04.311078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:04.311865 (+ 787us) server_negotiation.cc:407] Connection header received
0926 13:48:04.313045 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:04.313048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:04.313101 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:04.313197 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:04.314261 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.314762 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.315866 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.316079 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.316172 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:04.318588 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:04.318591 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:04.318593 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:04.318614 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:04.320094 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:04.320617 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:04.320620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:04.320622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:04.320714 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:04.321015 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:04.321019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:04.321020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:04.321229 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:04.321337 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:04.321944 (+ 607us) server_negotiation.cc:299] Negotiation successful
0926 13:48:04.322051 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 13:48:04.323242 29376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:04.311315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.67:43503)
0926 13:48:04.311740 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:04.311780 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:04.312856 (+ 1076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:04.313350 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:04.313357 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:04.313693 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:48:04.314129 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:04.314142 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.314869 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.314872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:04.315761 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:04.315770 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.316207 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.316211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:04.316367 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:04.316877 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:48:04.316891 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:48:04.318429 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:48:04.320200 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:04.320205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:04.320220 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:04.320528 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:04.320796 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:04.320798 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:04.320800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:04.320930 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:04.321355 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:48:04.321360 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:48:04.321680 (+ 320us) client_negotiation.cc:769] Sending connection context
0926 13:48:04.321898 (+ 218us) client_negotiation.cc:240] Negotiation successful
0926 13:48:04.322230 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":93,"threads_started":1}
I20260926 13:48:04.324405 29374 heartbeater.cc:347] Connected to a master server at 127.12.170.126:34267
I20260926 13:48:04.324668 29374 heartbeater.cc:464] Registering TS with master...
I20260926 13:48:04.325248 29374 heartbeater.cc:511] Master 127.12.170.126:34267 requested a full tablet report, sending...
I20260926 13:48:04.325448 29378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:04.317843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50472 (local address 127.12.170.126:34267)
0926 13:48:04.318117 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:04.318122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:04.318139 (+ 17us) server_negotiation.cc:407] Connection header received
0926 13:48:04.318267 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:04.318269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:04.318309 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:04.318390 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:04.319087 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.319627 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.320582 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.320908 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.320992 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:04.322460 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:04.322464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:04.322466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:04.322491 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:04.323863 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:04.324224 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:04.324227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:04.324229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:04.324298 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:04.324548 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:04.324551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:04.324553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:04.324781 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:04.325017 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:04.325147 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 13:48:04.325279 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":92,"threads_started":1}
I20260926 13:48:04.326265 28916 ts_manager.cc:194] Registered new tserver with Master: 59ec763eef0c4536a092003a70e34f06 (127.12.170.67:42981)
I20260926 13:48:04.326803 28916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.67@KRBTEST.COM'} at 127.12.170.67:43503
I20260926 13:48:04.327216 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:48:04.330870 29378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:04.327777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50484 (local address 127.12.170.126:34267)
0926 13:48:04.327884 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:04.327888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:04.328056 (+ 168us) server_negotiation.cc:407] Connection header received
0926 13:48:04.328207 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:04.328209 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:04.328286 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:04.328359 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:48:04.328569 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.329034 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.329869 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.330089 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.330172 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:04.330499 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:48:04.330607 (+ 108us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 13:48:04.330789 (+ 182us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50484: token expired
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 13:48:04.330935 29378 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50484: token expired
W20260926 13:48:04.331316 28845 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.170.126:34267)
I20260926 13:48:04.337670 29378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:04.331681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50490 (local address 127.12.170.126:34267)
0926 13:48:04.331768 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:04.331771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:04.331781 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:48:04.331859 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:04.331861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:04.331890 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:04.331945 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:04.332516 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.332970 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.333715 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.333901 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.333967 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:04.335085 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:04.335088 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:04.335089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:04.335107 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:04.336276 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:04.336646 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:04.336655 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:04.336656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:04.336718 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:04.336899 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:04.336901 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:04.336902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:04.337065 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:04.337161 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:04.337414 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 13:48:04.337532 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 13:48:04.339866 28916 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:48:04.342865 28916 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:48:04.349845 29038 tablet_service.cc:1467] Processing CreateTablet for tablet fee6c664a6764c9688178f53b2fbf4a4 (DEFAULT_TABLE table=test-table [id=84beeea55140496a822a95c1a531c9d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:48:04.350455 29038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee6c664a6764c9688178f53b2fbf4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:04.353174 29111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:04.348751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:34943 (local address 127.0.0.1:38710)
0926 13:48:04.349175 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:04.349189 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:04.349262 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:04.349820 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:04.349823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:04.349841 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:04.349928 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:04.349933 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.351075 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.351078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:04.352030 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:04.352039 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.352805 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.352808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:04.352889 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:04.352909 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 13:48:04.352979 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 13:48:04.353050 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347}
I20260926 13:48:04.353207 29384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:04.348929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38710 (local address 127.12.170.66:34943)
0926 13:48:04.349416 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:04.349427 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:04.349449 (+ 22us) server_negotiation.cc:407] Connection header received
0926 13:48:04.349522 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:04.349527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:04.349687 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:04.349829 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:04.350013 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.350950 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.352149 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.352687 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.352780 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:04.352915 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:04.353004 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 13:48:04.353084 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":125,"threads_started":1}
I20260926 13:48:04.353235 28996 tablet_bootstrap.cc:492] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912: Bootstrap starting.
I20260926 13:48:04.354025 29383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:04.348959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:42981 (local address 127.0.0.1:38002)
0926 13:48:04.349431 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:04.349442 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:04.349514 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:04.350190 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:04.350193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:04.350212 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:04.350307 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:04.350313 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.351565 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.351569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:04.352649 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:04.352660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.353673 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.353677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:04.353770 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:04.353793 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 13:48:04.353839 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 13:48:04.353902 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":97,"threads_started":1}
I20260926 13:48:04.354058 28996 tablet_bootstrap.cc:654] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:04.354609 29385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:04.349180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38002 (local address 127.12.170.67:42981)
0926 13:48:04.349733 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:04.349744 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:04.349764 (+ 20us) server_negotiation.cc:407] Connection header received
0926 13:48:04.349833 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:04.349837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:04.350020 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:04.350207 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:04.350413 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.351447 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.352791 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.353541 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.354212 (+ 671us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:04.354344 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:04.354393 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 13:48:04.354477 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":153,"threads_started":1}
I20260926 13:48:04.355172 28996 tablet_bootstrap.cc:492] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912: No bootstrap required, opened a new log
I20260926 13:48:04.355247 28996 ts_tablet_manager.cc:1397] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 13:48:04.355947 28996 raft_consensus.cc:348] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:48:04.356068 28996 raft_consensus.cc:374] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:04.356105 28996 raft_consensus.cc:729] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45aabb04e4564deda00644b409c8a912, State: Initialized, Role: FOLLOWER
I20260926 13:48:04.356222 28996 consensus_queue.cc:260] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:48:04.356464 28996 ts_tablet_manager.cc:1428] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:48:04.356340 29176 tablet_service.cc:1467] Processing CreateTablet for tablet fee6c664a6764c9688178f53b2fbf4a4 (DEFAULT_TABLE table=test-table [id=84beeea55140496a822a95c1a531c9d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:48:04.357354 29176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee6c664a6764c9688178f53b2fbf4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:04.357328 29308 tablet_service.cc:1467] Processing CreateTablet for tablet fee6c664a6764c9688178f53b2fbf4a4 (DEFAULT_TABLE table=test-table [id=84beeea55140496a822a95c1a531c9d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:48:04.358143 29308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee6c664a6764c9688178f53b2fbf4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:04.363829 29387 tablet_bootstrap.cc:492] T fee6c664a6764c9688178f53b2fbf4a4 P df30cd17e99f4882b7db22578c23543f: Bootstrap starting.
I20260926 13:48:04.363853 29388 tablet_bootstrap.cc:492] T fee6c664a6764c9688178f53b2fbf4a4 P 59ec763eef0c4536a092003a70e34f06: Bootstrap starting.
I20260926 13:48:04.365952 29388 tablet_bootstrap.cc:654] T fee6c664a6764c9688178f53b2fbf4a4 P 59ec763eef0c4536a092003a70e34f06: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:04.366039 29387 tablet_bootstrap.cc:654] T fee6c664a6764c9688178f53b2fbf4a4 P df30cd17e99f4882b7db22578c23543f: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:04.366736 29388 log.cc:824] T fee6c664a6764c9688178f53b2fbf4a4 P 59ec763eef0c4536a092003a70e34f06: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:04.366842 29387 log.cc:824] T fee6c664a6764c9688178f53b2fbf4a4 P df30cd17e99f4882b7db22578c23543f: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:04.368466 29387 tablet_bootstrap.cc:492] T fee6c664a6764c9688178f53b2fbf4a4 P df30cd17e99f4882b7db22578c23543f: No bootstrap required, opened a new log
I20260926 13:48:04.368476 29388 tablet_bootstrap.cc:492] T fee6c664a6764c9688178f53b2fbf4a4 P 59ec763eef0c4536a092003a70e34f06: No bootstrap required, opened a new log
I20260926 13:48:04.368623 29388 ts_tablet_manager.cc:1397] T fee6c664a6764c9688178f53b2fbf4a4 P 59ec763eef0c4536a092003a70e34f06: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 13:48:04.368623 29387 ts_tablet_manager.cc:1397] T fee6c664a6764c9688178f53b2fbf4a4 P df30cd17e99f4882b7db22578c23543f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 13:48:04.371259 29388 raft_consensus.cc:348] T fee6c664a6764c9688178f53b2fbf4a4 P 59ec763eef0c4536a092003a70e34f06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:48:04.371268 29387 raft_consensus.cc:348] T fee6c664a6764c9688178f53b2fbf4a4 P df30cd17e99f4882b7db22578c23543f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:48:04.371495 29388 raft_consensus.cc:374] T fee6c664a6764c9688178f53b2fbf4a4 P 59ec763eef0c4536a092003a70e34f06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:04.371500 29387 raft_consensus.cc:374] T fee6c664a6764c9688178f53b2fbf4a4 P df30cd17e99f4882b7db22578c23543f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:04.371547 29387 raft_consensus.cc:729] T fee6c664a6764c9688178f53b2fbf4a4 P df30cd17e99f4882b7db22578c23543f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df30cd17e99f4882b7db22578c23543f, State: Initialized, Role: FOLLOWER
I20260926 13:48:04.371551 29388 raft_consensus.cc:729] T fee6c664a6764c9688178f53b2fbf4a4 P 59ec763eef0c4536a092003a70e34f06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59ec763eef0c4536a092003a70e34f06, State: Initialized, Role: FOLLOWER
I20260926 13:48:04.371985 29388 consensus_queue.cc:260] T fee6c664a6764c9688178f53b2fbf4a4 P 59ec763eef0c4536a092003a70e34f06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:48:04.371990 29387 consensus_queue.cc:260] T fee6c664a6764c9688178f53b2fbf4a4 P df30cd17e99f4882b7db22578c23543f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:48:04.372568 29374 heartbeater.cc:503] Master 127.12.170.126:34267 was elected leader, sending a full tablet report...
I20260926 13:48:04.372568 29242 heartbeater.cc:503] Master 127.12.170.126:34267 was elected leader, sending a full tablet report...
I20260926 13:48:04.372864 29388 ts_tablet_manager.cc:1428] T fee6c664a6764c9688178f53b2fbf4a4 P 59ec763eef0c4536a092003a70e34f06: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260926 13:48:04.373945 29387 ts_tablet_manager.cc:1428] T fee6c664a6764c9688178f53b2fbf4a4 P df30cd17e99f4882b7db22578c23543f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260926 13:48:04.376277 29243 tablet.cc:2369] T fee6c664a6764c9688178f53b2fbf4a4 P df30cd17e99f4882b7db22578c23543f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:48:04.420248 29105 tablet.cc:2369] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:48:04.452865 29110 raft_consensus.cc:482] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:48:04.453058 29110 raft_consensus.cc:504] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:48:04.454042 29110 leader_election.cc:288] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df30cd17e99f4882b7db22578c23543f (127.12.170.66:34943), 59ec763eef0c4536a092003a70e34f06 (127.12.170.67:42981)
I20260926 13:48:04.457805 29384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:04.454387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:47755 (local address 127.12.170.66:34943)
0926 13:48:04.454513 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:04.454517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:04.454661 (+ 144us) server_negotiation.cc:407] Connection header received
0926 13:48:04.454835 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:04.454838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:04.454894 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:04.454937 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:04.455287 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.455892 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.456935 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.457327 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.457427 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:04.457477 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:04.457620 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 13:48:04.457685 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 13:48:04.457827 29385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:04.454810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:47749 (local address 127.12.170.67:42981)
0926 13:48:04.454928 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:04.454932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:04.455012 (+ 80us) server_negotiation.cc:407] Connection header received
0926 13:48:04.455182 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:04.455185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:04.455225 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:04.455282 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:04.455511 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.456044 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.456989 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.457367 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.457462 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:04.457511 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:04.457652 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 13:48:04.457710 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 13:48:04.457834 29393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:04.454310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:34943 (local address 127.12.170.65:47755)
0926 13:48:04.454554 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:04.454570 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:04.454690 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:04.455053 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:04.455055 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:04.455071 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:04.455148 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:04.455152 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.455996 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.455999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:04.456817 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:04.456826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.457427 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.457431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:04.457510 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:04.457533 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 13:48:04.457593 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 13:48:04.457660 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":32,"thread_start_us":72,"threads_started":1}
I20260926 13:48:04.457861 29394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:04.454668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:42981 (local address 127.12.170.65:47749)
0926 13:48:04.454928 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:04.454942 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:04.455052 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:04.455304 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:04.455307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:04.455322 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:04.455420 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:04.455425 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.456139 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.456143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:04.456884 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:04.456892 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.457463 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.457467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:04.457549 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:04.457570 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 13:48:04.457633 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 13:48:04.457709 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":2,"thread_start_us":87,"threads_started":1}
I20260926 13:48:04.458495 29196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee6c664a6764c9688178f53b2fbf4a4" candidate_uuid: "45aabb04e4564deda00644b409c8a912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df30cd17e99f4882b7db22578c23543f" is_pre_election: true
I20260926 13:48:04.458740 29196 raft_consensus.cc:2454] T fee6c664a6764c9688178f53b2fbf4a4 P df30cd17e99f4882b7db22578c23543f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45aabb04e4564deda00644b409c8a912 in term 0.
I20260926 13:48:04.459153 28991 leader_election.cc:302] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45aabb04e4564deda00644b409c8a912, df30cd17e99f4882b7db22578c23543f; no voters:
I20260926 13:48:04.459319 29328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee6c664a6764c9688178f53b2fbf4a4" candidate_uuid: "45aabb04e4564deda00644b409c8a912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59ec763eef0c4536a092003a70e34f06" is_pre_election: true
I20260926 13:48:04.459400 29110 raft_consensus.cc:2781] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:48:04.459481 29110 raft_consensus.cc:482] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:48:04.459530 29110 raft_consensus.cc:3037] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:04.459641 29328 raft_consensus.cc:2454] T fee6c664a6764c9688178f53b2fbf4a4 P 59ec763eef0c4536a092003a70e34f06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45aabb04e4564deda00644b409c8a912 in term 0.
I20260926 13:48:04.460613 29110 raft_consensus.cc:504] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:48:04.460911 29110 leader_election.cc:288] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [CANDIDATE]: Term 1 election: Requested vote from peers df30cd17e99f4882b7db22578c23543f (127.12.170.66:34943), 59ec763eef0c4536a092003a70e34f06 (127.12.170.67:42981)
I20260926 13:48:04.461264 29196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee6c664a6764c9688178f53b2fbf4a4" candidate_uuid: "45aabb04e4564deda00644b409c8a912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df30cd17e99f4882b7db22578c23543f"
I20260926 13:48:04.461352 29328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee6c664a6764c9688178f53b2fbf4a4" candidate_uuid: "45aabb04e4564deda00644b409c8a912" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59ec763eef0c4536a092003a70e34f06"
I20260926 13:48:04.461424 29196 raft_consensus.cc:3037] T fee6c664a6764c9688178f53b2fbf4a4 P df30cd17e99f4882b7db22578c23543f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:04.461462 29328 raft_consensus.cc:3037] T fee6c664a6764c9688178f53b2fbf4a4 P 59ec763eef0c4536a092003a70e34f06 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:04.462462 29196 raft_consensus.cc:2454] T fee6c664a6764c9688178f53b2fbf4a4 P df30cd17e99f4882b7db22578c23543f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45aabb04e4564deda00644b409c8a912 in term 1.
I20260926 13:48:04.462630 29328 raft_consensus.cc:2454] T fee6c664a6764c9688178f53b2fbf4a4 P 59ec763eef0c4536a092003a70e34f06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45aabb04e4564deda00644b409c8a912 in term 1.
I20260926 13:48:04.462759 28991 leader_election.cc:302] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45aabb04e4564deda00644b409c8a912, df30cd17e99f4882b7db22578c23543f; no voters:
I20260926 13:48:04.462929 29110 raft_consensus.cc:2781] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:48:04.463109 29110 raft_consensus.cc:686] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 1 LEADER]: Becoming Leader. State: Replica: 45aabb04e4564deda00644b409c8a912, State: Running, Role: LEADER
I20260926 13:48:04.463361 29110 consensus_queue.cc:237] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:48:04.466356 28915 catalog_manager.cc:5495] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45aabb04e4564deda00644b409c8a912 (127.12.170.65). New cstate: current_term: 1 leader_uuid: "45aabb04e4564deda00644b409c8a912" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } health_report { overall_health: HEALTHY } } }
I20260926 13:48:04.476763 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:04.473329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51504 (local address 127.12.170.65:41083)
0926 13:48:04.473559 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:04.473567 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:04.473579 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:48:04.473642 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:04.473645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:04.473691 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:04.473788 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:48:04.474032 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.474563 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.475371 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:04.475596 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:04.475730 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:04.475966 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:48:04.476385 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:48:04.476484 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:04.476552 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 13:48:04.476604 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":75,"threads_started":1}
I20260926 13:48:04.480367 29196 raft_consensus.cc:1261] T fee6c664a6764c9688178f53b2fbf4a4 P df30cd17e99f4882b7db22578c23543f [term 1 FOLLOWER]: Refusing update from remote peer 45aabb04e4564deda00644b409c8a912: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:48:04.480387 29328 raft_consensus.cc:1261] T fee6c664a6764c9688178f53b2fbf4a4 P 59ec763eef0c4536a092003a70e34f06 [term 1 FOLLOWER]: Refusing update from remote peer 45aabb04e4564deda00644b409c8a912: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:48:04.480966 29395 consensus_queue.cc:1035] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:48:04.481153 29110 consensus_queue.cc:1035] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:48:04.489892 29401 mvcc.cc:205] Tried to move back new op lower bound from 7333603264425582592 to 7333603264364318720. Current Snapshot: MvccSnapshot[applied={T|T < 7333603264425582592}]
I20260926 13:48:04.490283 29400 mvcc.cc:205] Tried to move back new op lower bound from 7333603264425582592 to 7333603264364318720. Current Snapshot: MvccSnapshot[applied={T|T < 7333603264425582592}]
I20260926 13:48:04.495568 28915 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:48:04.495765 28915 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:48:04.497984 28915 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb: Sending DeleteTablet for 3 replicas of tablet fee6c664a6764c9688178f53b2fbf4a4
I20260926 13:48:04.498564 29176 tablet_service.cc:1514] Processing DeleteTablet for tablet fee6c664a6764c9688178f53b2fbf4a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:48:04 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:38710
I20260926 13:48:04.498747 29308 tablet_service.cc:1514] Processing DeleteTablet for tablet fee6c664a6764c9688178f53b2fbf4a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:48:04 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:38002
I20260926 13:48:04.498932 29038 tablet_service.cc:1514] Processing DeleteTablet for tablet fee6c664a6764c9688178f53b2fbf4a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:48:04 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:51500
I20260926 13:48:04.499239 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 28954
I20260926 13:48:04.499370 29408 tablet_replica.cc:330] T fee6c664a6764c9688178f53b2fbf4a4 P df30cd17e99f4882b7db22578c23543f: stopping tablet replica
I20260926 13:48:04.499918 29408 raft_consensus.cc:2229] T fee6c664a6764c9688178f53b2fbf4a4 P df30cd17e99f4882b7db22578c23543f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:48:04.500218 29408 raft_consensus.cc:2258] T fee6c664a6764c9688178f53b2fbf4a4 P df30cd17e99f4882b7db22578c23543f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:48:04.501633 29408 ts_tablet_manager.cc:1905] T fee6c664a6764c9688178f53b2fbf4a4 P df30cd17e99f4882b7db22578c23543f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:48:04.503836 29409 tablet_replica.cc:330] T fee6c664a6764c9688178f53b2fbf4a4 P 59ec763eef0c4536a092003a70e34f06: stopping tablet replica
I20260926 13:48:04.504087 29409 raft_consensus.cc:2229] T fee6c664a6764c9688178f53b2fbf4a4 P 59ec763eef0c4536a092003a70e34f06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:48:04.504339 29409 raft_consensus.cc:2258] T fee6c664a6764c9688178f53b2fbf4a4 P 59ec763eef0c4536a092003a70e34f06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:48:04.504401 29408 ts_tablet_manager.cc:1918] T fee6c664a6764c9688178f53b2fbf4a4 P df30cd17e99f4882b7db22578c23543f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:48:04.504496 29408 log.cc:1196] T fee6c664a6764c9688178f53b2fbf4a4 P df30cd17e99f4882b7db22578c23543f: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/wal/wals/fee6c664a6764c9688178f53b2fbf4a4
I20260926 13:48:04.504878 29408 ts_tablet_manager.cc:1939] T fee6c664a6764c9688178f53b2fbf4a4 P df30cd17e99f4882b7db22578c23543f: Deleting consensus metadata
I20260926 13:48:04.505496 29409 ts_tablet_manager.cc:1905] T fee6c664a6764c9688178f53b2fbf4a4 P 59ec763eef0c4536a092003a70e34f06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:48:04.506062 28901 catalog_manager.cc:4841] TS df30cd17e99f4882b7db22578c23543f (127.12.170.66:34943): tablet fee6c664a6764c9688178f53b2fbf4a4 (table test-table [id=84beeea55140496a822a95c1a531c9d5]) successfully deleted
W20260926 13:48:04.506986 28901 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.170.65:41083 (error 108)
I20260926 13:48:04.507046 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 29108
I20260926 13:48:04.508375 29409 ts_tablet_manager.cc:1918] T fee6c664a6764c9688178f53b2fbf4a4 P 59ec763eef0c4536a092003a70e34f06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:48:04.508451 29409 log.cc:1196] T fee6c664a6764c9688178f53b2fbf4a4 P 59ec763eef0c4536a092003a70e34f06: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/fee6c664a6764c9688178f53b2fbf4a4
I20260926 13:48:04.508744 29409 ts_tablet_manager.cc:1939] T fee6c664a6764c9688178f53b2fbf4a4 P 59ec763eef0c4536a092003a70e34f06: Deleting consensus metadata
I20260926 13:48:04.509739 28901 catalog_manager.cc:4841] TS 59ec763eef0c4536a092003a70e34f06 (127.12.170.67:42981): tablet fee6c664a6764c9688178f53b2fbf4a4 (table test-table [id=84beeea55140496a822a95c1a531c9d5]) successfully deleted
I20260926 13:48:04.512130 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 29245
I20260926 13:48:04.512421 29383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:04.512149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:41083 (local address 127.0.0.1:51506)
0926 13:48:04.512285 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:04.512347 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.65:41083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 13:48:04.512790 28901 catalog_manager.cc:4584] TS 45aabb04e4564deda00644b409c8a912 (127.12.170.65:41083): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fee6c664a6764c9688178f53b2fbf4a4: Network error: Client connection negotiation failed: client connection to 127.12.170.65:41083: connect: Connection refused (error 111)
I20260926 13:48:04.518833 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 28883
I20260926 13:48:24.524179 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:34267
--webserver_interface=127.12.170.126
--webserver_port=38535
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:34267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:48:24.628795 29412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:24.629045 29412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:24.629099 29412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:24.632992 29412 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 13:48:24.633038 29412 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 13:48:24.633059 29412 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 13:48:24.633075 29412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:48:24.633098 29412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:24.633122 29412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:24.633152 29412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:48:24.633178 29412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:48:24.638320 29412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:34267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:34267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=38535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:24.639257 29412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:24.640221 29412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:24.648209 29418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:24.648247 29417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:24.648348 29412 server_base.cc:1034] running on GCE node
W20260926 13:48:24.648263 29420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:24.648854 29412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:24.649857 29412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:24.651022 29412 hybrid_clock.cc:648] HybridClock initialized: now 1790430504651003 us; error 30 us; skew 500 ppm
Sep 26 13:48:24 dist-test-slave-hc1k krb5kdc[28336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430504, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:24.653517 29412 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:48:24.654341 29412 webserver.cc:460] Webserver started at http://127.12.170.126:38535/ using document root <none> and password file <none>
I20260926 13:48:24.654844 29412 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:24.654906 29412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:24.658540 29412 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:48:24.660523 29427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:24.661659 29412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:48:24.661767 29412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "6d408b64f6fa430295bfcf3ff8e058cb"
format_stamp: "Formatted at 2026-09-26 13:47:42 on dist-test-slave-hc1k"
server_key: "0d55963306c5444d8aa0d1a4d17bbca9"
server_key_iv: "d1db2af6bdc006f9d1d991b5b09f7180"
server_key_version: "encryptionkey@0"
I20260926 13:48:24.662271 29412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:24.676709 29412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:24.679245 29412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:24.679402 29412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:24.685894 29412 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:34267
I20260926 13:48:24.686153 29479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:34267 every 8 connection(s)
I20260926 13:48:24.686873 29412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:48:24.689883 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 29412
I20260926 13:48:24.690341 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:41083
--local_ip_for_outbound_sockets=127.12.170.65
--tserver_master_addrs=127.12.170.126:34267
--webserver_port=41031
--webserver_interface=127.12.170.65
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:48:24.691182 29480 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:48:24.692929 29480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb: Bootstrap starting.
I20260926 13:48:24.709391 29480 log.cc:824] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:24.720082 29480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:48:24.720410 29480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb: Bootstrap complete.
I20260926 13:48:24.724109 29480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d408b64f6fa430295bfcf3ff8e058cb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34267 } }
I20260926 13:48:24.724500 29480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d408b64f6fa430295bfcf3ff8e058cb, State: Initialized, Role: FOLLOWER
I20260926 13:48:24.724946 29480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d408b64f6fa430295bfcf3ff8e058cb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34267 } }
I20260926 13:48:24.725060 29480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:48:24.725122 29480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:48:24.725211 29480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [term 2 FOLLOWER]: Advancing to term 3
I20260926 13:48:24.726887 29480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d408b64f6fa430295bfcf3ff8e058cb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34267 } }
I20260926 13:48:24.727171 29480 leader_election.cc:302] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d408b64f6fa430295bfcf3ff8e058cb; no voters:
I20260926 13:48:24.727429 29480 leader_election.cc:288] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 13:48:24.727550 29484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [term 3 FOLLOWER]: Leader election won for term 3
I20260926 13:48:24.727814 29484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [term 3 LEADER]: Becoming Leader. State: Replica: 6d408b64f6fa430295bfcf3ff8e058cb, State: Running, Role: LEADER
I20260926 13:48:24.728132 29484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d408b64f6fa430295bfcf3ff8e058cb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34267 } }
I20260926 13:48:24.728487 29480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:48:24.729028 29484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d408b64f6fa430295bfcf3ff8e058cb. Latest consensus state: current_term: 3 leader_uuid: "6d408b64f6fa430295bfcf3ff8e058cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d408b64f6fa430295bfcf3ff8e058cb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34267 } } }
I20260926 13:48:24.729132 29484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:24.729521 29485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6d408b64f6fa430295bfcf3ff8e058cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d408b64f6fa430295bfcf3ff8e058cb" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 34267 } } }
I20260926 13:48:24.729669 29485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:24.733476 29499 catalog_manager.cc:1259] Loaded cluster ID: ca6bcd6339d44dd5bfea418a807307e3
I20260926 13:48:24.733529 29499 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb: loading cluster ID for follower catalog manager: success
I20260926 13:48:24.735863 29499 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6d408b64f6fa430295bfcf3ff8e058cb: acquiring CA information for follower catalog manager: success
F20260926 13:48:24.736959 29499 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc59677f9ad google::LogMessage::Fail() at ??:0
@ 0x7fc59678288a google::LogMessage::SendToLog() at ??:0
@ 0x7fc59677f4de google::LogMessage::Flush() at ??:0
@ 0x7fc596780a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc59944318d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc5a10e8989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7fc5a10c5023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7fc5a10c554b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7fc5a10be652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7fc5a10bd9e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7fc5a1104541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc5989c2512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc59a0066db start_thread at ??:0
@ 0x7fc59527d61f clone at ??:0
W20260926 13:48:24.810096 29482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:24.810309 29482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:24.810364 29482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:24.814232 29482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:24.814280 29482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:24.814350 29482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:48:24.819272 29482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:41083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=41031
--tserver_master_addrs=127.12.170.126:34267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:24.820206 29482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:24.821105 29482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:24.829739 29520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:24.829767 29523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:24.829769 29521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:24.830080 29482 server_base.cc:1034] running on GCE node
I20260926 13:48:24.830410 29482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:24.830956 29482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:24.832113 29482 hybrid_clock.cc:648] HybridClock initialized: now 1790430504832091 us; error 43 us; skew 500 ppm
Sep 26 13:48:24 dist-test-slave-hc1k krb5kdc[28336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430504, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:24.834604 29482 init.cc:377] Logged in from keytab as kudu/127.12.170.65@KRBTEST.COM (short username kudu)
I20260926 13:48:24.835424 29482 webserver.cc:460] Webserver started at http://127.12.170.65:41031/ using document root <none> and password file <none>
I20260926 13:48:24.835970 29482 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:24.836027 29482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:24.839494 29482 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:48:24.841419 29530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:24.842556 29482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:48:24.842643 29482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "45aabb04e4564deda00644b409c8a912"
format_stamp: "Formatted at 2026-09-26 13:47:42 on dist-test-slave-hc1k"
server_key: "9b887b3e6c8a71e9bd4fb92dbdaca565"
server_key_iv: "f59e86eb5f33d71621bd86a0949e826b"
server_key_version: "encryptionkey@0"
I20260926 13:48:24.843135 29482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:24.862586 29482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:24.865476 29482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:24.865624 29482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:24.866153 29482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:48:24.867195 29537 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 13:48:24.872216 29482 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 13:48:24.872280 29482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260926 13:48:24.872344 29482 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 13:48:24.873934 29482 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 13:48:24.873975 29482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 13:48:24.874286 29537 tablet_bootstrap.cc:492] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912: Bootstrap starting.
I20260926 13:48:24.883193 29482 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:41083
I20260926 13:48:24.883255 29644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:41083 every 8 connection(s)
I20260926 13:48:24.884084 29482 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:48:24.886440 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 29482
I20260926 13:48:24.886806 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:34943
--local_ip_for_outbound_sockets=127.12.170.66
--tserver_master_addrs=127.12.170.126:34267
--webserver_port=37053
--webserver_interface=127.12.170.66
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:48:24.895061 29537 log.cc:824] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:24.901913 29537 tablet_bootstrap.cc:492] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:48:24.902244 29537 tablet_bootstrap.cc:492] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912: Bootstrap complete.
I20260926 13:48:24.903110 29537 ts_tablet_manager.cc:1397] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.011s
I20260926 13:48:24.905551 29537 raft_consensus.cc:348] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:48:24.905954 29537 raft_consensus.cc:729] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45aabb04e4564deda00644b409c8a912, State: Initialized, Role: FOLLOWER
I20260926 13:48:24.906399 29537 consensus_queue.cc:260] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:48:24.907274 29537 ts_tablet_manager.cc:1428] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912: Time spent starting tablet: real 0.004s user 0.006s sys 0.001s
W20260926 13:48:24.950003 29647 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:48:24.885778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.65:60735)
0926 13:48:24.886219 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:24.887421 (+ 1202us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:24.888366 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:24.949288 (+ 60922us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":99,"threads_started":1}
W20260926 13:48:24.950433 29645 heartbeater.cc:644] Failed to heartbeat to 127.12.170.126:34267 (0 consecutive failures): Network error: Failed to ping master at 127.12.170.126:34267: Client connection negotiation failed: client connection to 127.12.170.126:34267: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 13:48:24.951117 29647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:24.950890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.65:44807)
0926 13:48:24.951006 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:24.951064 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 13:48:24.951787 29647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:24.951597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.65:40269)
0926 13:48:24.951700 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:24.951741 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 13:48:24.951934 29645 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:48:25.011911 29648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:25.012118 29648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:25.012156 29648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:25.016013 29648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:25.016057 29648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:25.016111 29648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:48:25.020927 29648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:34943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=37053
--tserver_master_addrs=127.12.170.126:34267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:25.021723 29648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:25.022574 29648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:48:25.031072 29648 server_base.cc:1034] running on GCE node
W20260926 13:48:25.031042 29658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:25.031028 29655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:25.031056 29656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:25.031569 29648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:25.032140 29648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:25.033298 29648 hybrid_clock.cc:648] HybridClock initialized: now 1790430505033284 us; error 25 us; skew 500 ppm
Sep 26 13:48:25 dist-test-slave-hc1k krb5kdc[28336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430505, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:25.035727 29648 init.cc:377] Logged in from keytab as kudu/127.12.170.66@KRBTEST.COM (short username kudu)
I20260926 13:48:25.036448 29648 webserver.cc:460] Webserver started at http://127.12.170.66:37053/ using document root <none> and password file <none>
I20260926 13:48:25.036897 29648 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:25.036948 29648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:25.040326 29648 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:48:25.042102 29665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:25.043251 29648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:48:25.043349 29648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "df30cd17e99f4882b7db22578c23543f"
format_stamp: "Formatted at 2026-09-26 13:47:43 on dist-test-slave-hc1k"
server_key: "391c83058789f24e3e67d28c92174810"
server_key_iv: "a551dfd45eec0013e972eca479ed0a6d"
server_key_version: "encryptionkey@0"
I20260926 13:48:25.043860 29648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:25.061865 29648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:25.064502 29648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:25.064632 29648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:25.065111 29648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:48:25.065806 29648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:48:25.065847 29648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:25.065883 29648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:48:25.065902 29648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:25.073565 29648 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:34943
I20260926 13:48:25.073592 29778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:34943 every 8 connection(s)
I20260926 13:48:25.074425 29648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:48:25.075861 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 29648
I20260926 13:48:25.076215 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:42981
--local_ip_for_outbound_sockets=127.12.170.67
--tserver_master_addrs=127.12.170.126:34267
--webserver_port=40893
--webserver_interface=127.12.170.67
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:41667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:48:25.078171 29781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:25.076331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.66:49879)
0926 13:48:25.076813 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:25.076933 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":106,"threads_started":1}
W20260926 13:48:25.078594 29779 heartbeater.cc:644] Failed to heartbeat to 127.12.170.126:34267 (0 consecutive failures): Network error: Failed to ping master at 127.12.170.126:34267: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
I20260926 13:48:25.079107 29781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:25.078912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.66:40105)
0926 13:48:25.079011 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:25.079057 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 13:48:25.079697 29781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:25.079509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.66:50599)
0926 13:48:25.079588 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:25.079624 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 13:48:25.079917 29779 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:48:25.180228 29782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:25.180445 29782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:25.180482 29782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:25.184401 29782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:25.184451 29782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:25.184509 29782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:48:25.189471 29782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:41667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:42981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=40893
--tserver_master_addrs=127.12.170.126:34267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:25.190340 29782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:25.191244 29782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:25.200047 29787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:25.200050 29790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:25.200040 29788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:25.200099 29782 server_base.cc:1034] running on GCE node
I20260926 13:48:25.200548 29782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:25.201078 29782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:25.202261 29782 hybrid_clock.cc:648] HybridClock initialized: now 1790430505202232 us; error 40 us; skew 500 ppm
Sep 26 13:48:25 dist-test-slave-hc1k krb5kdc[28336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430505, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:25.205440 29782 init.cc:377] Logged in from keytab as kudu/127.12.170.67@KRBTEST.COM (short username kudu)
I20260926 13:48:25.206235 29782 webserver.cc:460] Webserver started at http://127.12.170.67:40893/ using document root <none> and password file <none>
I20260926 13:48:25.206732 29782 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:25.206794 29782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:25.210321 29782 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:48:25.212282 29797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:25.213392 29782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:48:25.213495 29782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "59ec763eef0c4536a092003a70e34f06"
format_stamp: "Formatted at 2026-09-26 13:47:43 on dist-test-slave-hc1k"
server_key: "f15f1e5f70913431aaab7aa764669a84"
server_key_iv: "77dec03e1497067074d8041b5eda4737"
server_key_version: "encryptionkey@0"
I20260926 13:48:25.213982 29782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:25.233963 29782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:25.236466 29782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:25.236615 29782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:25.237116 29782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:48:25.237854 29782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:48:25.237897 29782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:25.237944 29782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:48:25.237972 29782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:25.246075 29782 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:42981
I20260926 13:48:25.246083 29910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:42981 every 8 connection(s)
I20260926 13:48:25.246987 29782 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:48:25.250427 29913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:25.248992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.67:52897)
0926 13:48:25.249390 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:25.249769 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":96,"threads_started":1}
W20260926 13:48:25.250833 29911 heartbeater.cc:644] Failed to heartbeat to 127.12.170.126:34267 (0 consecutive failures): Network error: Failed to ping master at 127.12.170.126:34267: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
I20260926 13:48:25.251391 29913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:25.251137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.67:60619)
0926 13:48:25.251285 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:25.251331 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 13:48:25.251959 29913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:25.251791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.67:46087)
0926 13:48:25.251873 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:25.251912 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 13:48:25.252110 29911 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 13:48:25.252193 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 29782
I20260926 13:48:25.952836 29915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:25.952382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.65:51087)
0926 13:48:25.952644 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:25.952728 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":70,"threads_started":1}
I20260926 13:48:26.080775 29916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:26.080401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.66:38323)
0926 13:48:26.080628 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:26.080685 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":62,"threads_started":1}
I20260926 13:48:26.252903 29917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:26.252560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.67:56315)
0926 13:48:26.252764 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:26.252812 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":64,"threads_started":1}
I20260926 13:48:26.459527 29918 raft_consensus.cc:482] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:48:26.459766 29918 raft_consensus.cc:504] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:48:26.460835 29918 leader_election.cc:288] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df30cd17e99f4882b7db22578c23543f (127.12.170.66:34943), 59ec763eef0c4536a092003a70e34f06 (127.12.170.67:42981)
Sep 26 13:48:26 dist-test-slave-hc1k krb5kdc[28336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430504, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for kudu/127.12.170.67@KRBTEST.COM
Sep 26 13:48:26 dist-test-slave-hc1k krb5kdc[28336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430504, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for kudu/127.12.170.66@KRBTEST.COM
I20260926 13:48:26.474068 29921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:26.461412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:42981 (local address 127.12.170.65:36175)
0926 13:48:26.461632 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:26.461646 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:26.461753 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:26.463104 (+ 1351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:26.463110 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:26.463610 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:48:26.464111 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:26.464125 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:26.465371 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:26.465374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:26.466537 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:26.466548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:26.467208 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:26.467211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:26.467414 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:26.468039 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:48:26.468060 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:48:26.469678 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:48:26.471560 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:26.471565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:26.471579 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:26.471913 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:26.472255 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:26.472258 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:26.472259 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:26.472426 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:26.473083 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:48:26.473089 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:48:26.473362 (+ 273us) client_negotiation.cc:769] Sending connection context
0926 13:48:26.473525 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 13:48:26.473866 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":247,"spinlock_wait_cycles":12672,"thread_start_us":68,"threads_started":1}
I20260926 13:48:26.474066 29920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:26.461196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:35371 (local address 127.12.170.66:34943)
0926 13:48:26.461605 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:26.461631 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:26.461696 (+ 65us) server_negotiation.cc:407] Connection header received
0926 13:48:26.462393 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:26.462399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:26.462738 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:26.462972 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:26.464268 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:26.465233 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:26.466642 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:26.467087 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:26.467225 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:26.470159 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:26.470164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:26.470174 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:26.470205 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:26.471897 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:26.472350 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:26.472355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:26.472360 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:26.472479 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:26.472766 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:26.472769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:26.472770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:26.473171 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:26.473300 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:26.473650 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 13:48:26.473906 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":87,"threads_started":1}
I20260926 13:48:26.474088 29922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:26.461616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:36175 (local address 127.12.170.67:42981)
0926 13:48:26.461869 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:26.461890 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:26.461922 (+ 32us) server_negotiation.cc:407] Connection header received
0926 13:48:26.462530 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:26.462535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:26.462838 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:26.463122 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:26.464274 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:26.465249 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:26.466655 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:26.467091 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:26.467240 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:26.469911 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:26.469916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:26.469927 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:26.469952 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:26.471440 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:26.472003 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:26.472008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:26.472014 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:26.472165 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:26.472517 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:26.472521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:26.472524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:26.472960 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:26.473055 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:26.473678 (+ 623us) server_negotiation.cc:299] Negotiation successful
0926 13:48:26.473913 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":61,"threads_started":1}
I20260926 13:48:26.474090 29919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:26.461027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:34943 (local address 127.12.170.65:35371)
0926 13:48:26.461288 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:26.461304 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:26.461411 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:26.463104 (+ 1693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:26.463110 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:26.463616 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:48:26.464113 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:26.464126 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:26.465381 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:26.465384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:26.466532 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:26.466543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:26.467206 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:26.467211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:26.467419 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:26.468083 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:48:26.468095 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:48:26.469997 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:48:26.472003 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:26.472006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:26.472008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:26.472257 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:26.472568 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:26.472572 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:26.472573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:26.472651 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:26.473286 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:48:26.473289 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:48:26.473368 (+ 79us) client_negotiation.cc:769] Sending connection context
0926 13:48:26.473525 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 13:48:26.473935 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":361,"spinlock_wait_cycles":29696,"thread_start_us":72,"threads_started":1}
W20260926 13:48:26.476701 29532 leader_election.cc:341] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59ec763eef0c4536a092003a70e34f06 (127.12.170.67:42981): Not found: Tablet not found: fee6c664a6764c9688178f53b2fbf4a4
W20260926 13:48:26.476933 29532 leader_election.cc:341] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df30cd17e99f4882b7db22578c23543f (127.12.170.66:34943): Not found: Tablet not found: fee6c664a6764c9688178f53b2fbf4a4
I20260926 13:48:26.477005 29532 leader_election.cc:302] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45aabb04e4564deda00644b409c8a912; no voters: 59ec763eef0c4536a092003a70e34f06, df30cd17e99f4882b7db22578c23543f
I20260926 13:48:26.477255 29918 raft_consensus.cc:2726] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:48:26.953747 29919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:26.953454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.65:36691)
0926 13:48:26.953596 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:26.953660 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 13:48:27.081725 29923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:27.081387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.66:47951)
0926 13:48:27.081586 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:27.081638 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":63,"threads_started":1}
I20260926 13:48:27.253834 29924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:27.253494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.67:53979)
0926 13:48:27.253698 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:27.253759 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":64,"threads_started":1}
I20260926 13:48:27.954804 29925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:27.954397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.65:45927)
0926 13:48:27.954628 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:27.954682 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":67,"threads_started":1}
I20260926 13:48:28.082832 29926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:28.082387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.66:51439)
0926 13:48:28.082684 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:28.082740 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":75,"threads_started":1}
I20260926 13:48:28.254788 29927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:28.254429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.67:36909)
0926 13:48:28.254658 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:28.254708 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":79,"threads_started":1}
I20260926 13:48:28.467376 29928 raft_consensus.cc:482] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:48:28.467499 29928 raft_consensus.cc:504] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:48:28.467906 29928 leader_election.cc:288] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df30cd17e99f4882b7db22578c23543f (127.12.170.66:34943), 59ec763eef0c4536a092003a70e34f06 (127.12.170.67:42981)
W20260926 13:48:28.468627 29532 leader_election.cc:341] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df30cd17e99f4882b7db22578c23543f (127.12.170.66:34943): Not found: Tablet not found: fee6c664a6764c9688178f53b2fbf4a4
W20260926 13:48:28.468743 29532 leader_election.cc:341] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59ec763eef0c4536a092003a70e34f06 (127.12.170.67:42981): Not found: Tablet not found: fee6c664a6764c9688178f53b2fbf4a4
I20260926 13:48:28.468817 29532 leader_election.cc:302] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45aabb04e4564deda00644b409c8a912; no voters: 59ec763eef0c4536a092003a70e34f06, df30cd17e99f4882b7db22578c23543f
I20260926 13:48:28.468958 29928 raft_consensus.cc:2726] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:48:28.955885 29929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:28.955450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.65:47563)
0926 13:48:28.955724 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:28.955778 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":67,"threads_started":1}
I20260926 13:48:29.083889 29930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:29.083482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.66:39339)
0926 13:48:29.083747 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:29.083805 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":66,"threads_started":1}
I20260926 13:48:29.255853 29931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:29.255424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.67:46983)
0926 13:48:29.255720 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:29.255772 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
I20260926 13:48:29.957057 29932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:29.956644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.65:55995)
0926 13:48:29.956916 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:29.956965 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":69,"threads_started":1}
I20260926 13:48:30.084962 29933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:30.084572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.66:48191)
0926 13:48:30.084809 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:30.084863 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":65,"threads_started":1}
I20260926 13:48:30.256994 29934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:30.256550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.67:42685)
0926 13:48:30.256801 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:30.256888 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":77,"threads_started":1}
I20260926 13:48:30.958120 29935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:30.957732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.65:38385)
0926 13:48:30.957965 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:30.958023 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":70,"threads_started":1}
I20260926 13:48:31.086012 29936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:31.085619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.66:57197)
0926 13:48:31.085872 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:31.085926 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
I20260926 13:48:31.149945 29937 raft_consensus.cc:482] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:48:31.150098 29937 raft_consensus.cc:504] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:48:31.150494 29937 leader_election.cc:288] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df30cd17e99f4882b7db22578c23543f (127.12.170.66:34943), 59ec763eef0c4536a092003a70e34f06 (127.12.170.67:42981)
W20260926 13:48:31.151132 29532 leader_election.cc:341] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df30cd17e99f4882b7db22578c23543f (127.12.170.66:34943): Not found: Tablet not found: fee6c664a6764c9688178f53b2fbf4a4
W20260926 13:48:31.151257 29532 leader_election.cc:341] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59ec763eef0c4536a092003a70e34f06 (127.12.170.67:42981): Not found: Tablet not found: fee6c664a6764c9688178f53b2fbf4a4
I20260926 13:48:31.151295 29532 leader_election.cc:302] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45aabb04e4564deda00644b409c8a912; no voters: 59ec763eef0c4536a092003a70e34f06, df30cd17e99f4882b7db22578c23543f
I20260926 13:48:31.151414 29937 raft_consensus.cc:2726] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:48:31.258049 29938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:31.257674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.67:38901)
0926 13:48:31.257915 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:31.257966 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
I20260926 13:48:31.959226 29939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:31.958830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.65:41715)
0926 13:48:31.959076 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:31.959131 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":81,"threads_started":1}
I20260926 13:48:32.087085 29940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:32.086650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.66:58399)
0926 13:48:32.086950 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:32.087004 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":86,"threads_started":1}
I20260926 13:48:32.259054 29941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:32.258659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.67:53445)
0926 13:48:32.258926 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:32.258975 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":74,"threads_started":1}
I20260926 13:48:32.960331 29942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:32.959940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.65:60421)
0926 13:48:32.960179 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:32.960236 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
I20260926 13:48:33.088222 29943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:33.087828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.66:52077)
0926 13:48:33.088076 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:33.088128 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":88,"threads_started":1}
I20260926 13:48:33.260114 29944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:33.259734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.67:57123)
0926 13:48:33.259965 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:33.260022 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
I20260926 13:48:33.961383 29945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:33.960967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.65:49361)
0926 13:48:33.961236 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:33.961291 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":75,"threads_started":1}
I20260926 13:48:34.089259 29946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:34.088834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.66:41669)
0926 13:48:34.089078 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:34.089137 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":66,"threads_started":1}
I20260926 13:48:34.261093 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:34.260756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.67:35217)
0926 13:48:34.260963 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:34.261011 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":66,"threads_started":1}
I20260926 13:48:34.358409 29948 raft_consensus.cc:482] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:48:34.358529 29948 raft_consensus.cc:504] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:48:34.358917 29948 leader_election.cc:288] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df30cd17e99f4882b7db22578c23543f (127.12.170.66:34943), 59ec763eef0c4536a092003a70e34f06 (127.12.170.67:42981)
W20260926 13:48:34.359752 29532 leader_election.cc:341] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59ec763eef0c4536a092003a70e34f06 (127.12.170.67:42981): Not found: Tablet not found: fee6c664a6764c9688178f53b2fbf4a4
W20260926 13:48:34.359867 29532 leader_election.cc:341] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df30cd17e99f4882b7db22578c23543f (127.12.170.66:34943): Not found: Tablet not found: fee6c664a6764c9688178f53b2fbf4a4
I20260926 13:48:34.359917 29532 leader_election.cc:302] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45aabb04e4564deda00644b409c8a912; no voters: 59ec763eef0c4536a092003a70e34f06, df30cd17e99f4882b7db22578c23543f
I20260926 13:48:34.360040 29948 raft_consensus.cc:2726] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:48:34.962514 29949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:34.962116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.65:51817)
0926 13:48:34.962365 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:34.962420 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":66,"threads_started":1}
I20260926 13:48:35.090301 29950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:35.089931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.66:58407)
0926 13:48:35.090161 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:35.090215 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":67,"threads_started":1}
I20260926 13:48:35.262151 29951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:35.261764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.67:56827)
0926 13:48:35.262006 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:35.262062 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":78,"threads_started":1}
I20260926 13:48:35.963541 29952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:35.963153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.65:36347)
0926 13:48:35.963392 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:35.963444 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260926 13:48:36.091377 29953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:36.090964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.66:60349)
0926 13:48:36.091207 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:36.091289 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":79,"threads_started":1}
I20260926 13:48:36.263198 29954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:36.262820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.67:43815)
0926 13:48:36.263060 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:36.263117 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
I20260926 13:48:36.964603 29955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:36.964181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.65:41085)
0926 13:48:36.964441 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:36.964503 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":67,"threads_started":1}
I20260926 13:48:37.092392 29956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:37.092031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.66:42637)
0926 13:48:37.092253 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:37.092305 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":68,"threads_started":1}
I20260926 13:48:37.264259 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:37.263885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.67:49169)
0926 13:48:37.264110 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:37.264163 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":66,"threads_started":1}
I20260926 13:48:37.965669 29958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:37.965268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.65:39667)
0926 13:48:37.965522 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:37.965576 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":82,"threads_started":1}
I20260926 13:48:38.051430 29959 raft_consensus.cc:482] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:48:38.051548 29959 raft_consensus.cc:504] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df30cd17e99f4882b7db22578c23543f" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 34943 } } peers { permanent_uuid: "59ec763eef0c4536a092003a70e34f06" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42981 } } peers { permanent_uuid: "45aabb04e4564deda00644b409c8a912" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 41083 } }
I20260926 13:48:38.051929 29959 leader_election.cc:288] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df30cd17e99f4882b7db22578c23543f (127.12.170.66:34943), 59ec763eef0c4536a092003a70e34f06 (127.12.170.67:42981)
W20260926 13:48:38.052662 29532 leader_election.cc:341] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer df30cd17e99f4882b7db22578c23543f (127.12.170.66:34943): Not found: Tablet not found: fee6c664a6764c9688178f53b2fbf4a4
W20260926 13:48:38.052802 29532 leader_election.cc:341] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59ec763eef0c4536a092003a70e34f06 (127.12.170.67:42981): Not found: Tablet not found: fee6c664a6764c9688178f53b2fbf4a4
I20260926 13:48:38.052851 29532 leader_election.cc:302] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45aabb04e4564deda00644b409c8a912; no voters: 59ec763eef0c4536a092003a70e34f06, df30cd17e99f4882b7db22578c23543f
I20260926 13:48:38.053005 29959 raft_consensus.cc:2726] T fee6c664a6764c9688178f53b2fbf4a4 P 45aabb04e4564deda00644b409c8a912 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:48:38.093361 29960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:38.092979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.66:35113)
0926 13:48:38.093231 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:38.093281 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":90,"threads_started":1}
I20260926 13:48:38.265293 29961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:38.264898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.67:52815)
0926 13:48:38.265148 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:38.265202 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":73,"threads_started":1}
I20260926 13:48:38.966728 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:38.966318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.65:47595)
0926 13:48:38.966582 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:38.966637 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
I20260926 13:48:39.094374 29963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:39.093982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.66:39463)
0926 13:48:39.094222 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:39.094280 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":72,"threads_started":1}
I20260926 13:48:39.266325 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:39.265945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.67:49373)
0926 13:48:39.266177 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:39.266228 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":79,"threads_started":1}
I20260926 13:48:39.967981 29965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:39.967576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.65:59921)
0926 13:48:39.967837 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:39.967889 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":92,"threads_started":1}
I20260926 13:48:40.095503 29966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:40.095131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.66:57691)
0926 13:48:40.095361 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:40.095414 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":65,"threads_started":1}
I20260926 13:48:40.260056 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 29482
I20260926 13:48:40.264999 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 29648
I20260926 13:48:40.267427 29967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:40.267059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:34267 (local address 127.12.170.67:60845)
0926 13:48:40.267281 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:40.267330 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.126:34267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":63,"threads_started":1}
I20260926 13:48:40.269500 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 29782
2026-09-26T13:48:40Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58808 ms)
[----------] 30 tests from SecurityITest (228324 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:48:40 dist-test-slave-hc1k krb5kdc[29970](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:48:40 dist-test-slave-hc1k krb5kdc[29970](info): set up 2 sockets
Sep 26 13:48:40 dist-test-slave-hc1k krb5kdc[29970](info): commencing operation
krb5kdc: starting...
W20260926 13:48:41.231959 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:48:41 dist-test-slave-hc1k krb5kdc[29970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:48:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:48:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:48:41.342612 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:36577
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:45407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:36577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:48:41.445308 29983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:41.445530 29983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:41.445567 29983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:41.449489 29983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:48:41.449534 29983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:41.449551 29983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:41.449566 29983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:48:41.449594 29983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:48:41.454551 29983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:45407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:36577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:36577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:41.455440 29983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:41.456357 29983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:41.463948 29989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:41.463985 29988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:41.464052 29983 server_base.cc:1034] running on GCE node
W20260926 13:48:41.464005 29991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:41.464720 29983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:41.465724 29983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:41.466892 29983 hybrid_clock.cc:648] HybridClock initialized: now 1790430521466873 us; error 32 us; skew 500 ppm
Sep 26 13:48:41 dist-test-slave-hc1k krb5kdc[29970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430521, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:41.469408 29983 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:48:41.470165 29983 webserver.cc:460] Webserver started at http://127.12.170.126:35149/ using document root <none> and password file <none>
I20260926 13:48:41.470651 29983 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:41.470703 29983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:41.470863 29983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:41.472832 29983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "56d5e9c47b914483a14f573caa792c32"
format_stamp: "Formatted at 2026-09-26 13:48:41 on dist-test-slave-hc1k"
server_key: "2a034aa29d947fa0e3f881c19dc4a16b"
server_key_iv: "cdc6f085ff6c7a28e8e3e27edc72423a"
server_key_version: "encryptionkey@0"
I20260926 13:48:41.473304 29983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "56d5e9c47b914483a14f573caa792c32"
format_stamp: "Formatted at 2026-09-26 13:48:41 on dist-test-slave-hc1k"
server_key: "2a034aa29d947fa0e3f881c19dc4a16b"
server_key_iv: "cdc6f085ff6c7a28e8e3e27edc72423a"
server_key_version: "encryptionkey@0"
I20260926 13:48:41.476701 29983 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:48:41.478801 29998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:41.479949 29983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:48:41.480050 29983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "56d5e9c47b914483a14f573caa792c32"
format_stamp: "Formatted at 2026-09-26 13:48:41 on dist-test-slave-hc1k"
server_key: "2a034aa29d947fa0e3f881c19dc4a16b"
server_key_iv: "cdc6f085ff6c7a28e8e3e27edc72423a"
server_key_version: "encryptionkey@0"
I20260926 13:48:41.480134 29983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:41.498773 29983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:41.501240 29983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:41.501403 29983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:41.507671 29983 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:36577
I20260926 13:48:41.507696 30050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:36577 every 8 connection(s)
I20260926 13:48:41.508510 29983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:48:41.511005 30051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:41.516105 30051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d5e9c47b914483a14f573caa792c32: Bootstrap starting.
I20260926 13:48:41.517707 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 29983
I20260926 13:48:41.517831 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:48:41.518112 12969 external_mini_cluster.cc:1428] Setting key 00296088b7be558ac9d2abebb7ee8b41
I20260926 13:48:41.518601 30051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56d5e9c47b914483a14f573caa792c32: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:41.519419 30051 log.cc:824] T 00000000000000000000000000000000 P 56d5e9c47b914483a14f573caa792c32: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:41.521296 30051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d5e9c47b914483a14f573caa792c32: No bootstrap required, opened a new log
I20260926 13:48:41.525110 30051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56d5e9c47b914483a14f573caa792c32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d5e9c47b914483a14f573caa792c32" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36577 } }
I20260926 13:48:41.525372 30051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56d5e9c47b914483a14f573caa792c32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:41.525435 30051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56d5e9c47b914483a14f573caa792c32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d5e9c47b914483a14f573caa792c32, State: Initialized, Role: FOLLOWER
I20260926 13:48:41.525929 30051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56d5e9c47b914483a14f573caa792c32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d5e9c47b914483a14f573caa792c32" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36577 } }
I20260926 13:48:41.526082 30051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56d5e9c47b914483a14f573caa792c32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:48:41.526149 30051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56d5e9c47b914483a14f573caa792c32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:48:41.526247 30051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56d5e9c47b914483a14f573caa792c32 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 13:48:41 dist-test-slave-hc1k krb5kdc[29970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:48:41.527081 30051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56d5e9c47b914483a14f573caa792c32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d5e9c47b914483a14f573caa792c32" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36577 } }
I20260926 13:48:41.527377 30051 leader_election.cc:302] T 00000000000000000000000000000000 P 56d5e9c47b914483a14f573caa792c32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56d5e9c47b914483a14f573caa792c32; no voters:
I20260926 13:48:41.527644 30051 leader_election.cc:288] T 00000000000000000000000000000000 P 56d5e9c47b914483a14f573caa792c32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:48:41.527737 30056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56d5e9c47b914483a14f573caa792c32 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:48:41.527925 30056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56d5e9c47b914483a14f573caa792c32 [term 1 LEADER]: Becoming Leader. State: Replica: 56d5e9c47b914483a14f573caa792c32, State: Running, Role: LEADER
I20260926 13:48:41.528252 30056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56d5e9c47b914483a14f573caa792c32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d5e9c47b914483a14f573caa792c32" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36577 } }
I20260926 13:48:41.528868 30051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56d5e9c47b914483a14f573caa792c32 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:48:41.529374 30056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d5e9c47b914483a14f573caa792c32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56d5e9c47b914483a14f573caa792c32. Latest consensus state: current_term: 1 leader_uuid: "56d5e9c47b914483a14f573caa792c32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d5e9c47b914483a14f573caa792c32" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36577 } } }
I20260926 13:48:41.530081 30056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d5e9c47b914483a14f573caa792c32 [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:41.529975 30057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d5e9c47b914483a14f573caa792c32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56d5e9c47b914483a14f573caa792c32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d5e9c47b914483a14f573caa792c32" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 36577 } } }
I20260926 13:48:41.530998 30057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d5e9c47b914483a14f573caa792c32 [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:41.531268 30068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:48:41.532313 30054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:41.519225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55468 (local address 127.12.170.126:36577)
0926 13:48:41.519626 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:41.519642 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:41.519716 (+ 74us) server_negotiation.cc:407] Connection header received
0926 13:48:41.520470 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:41.520490 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:41.520807 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:41.521084 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:41.521920 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:41.522841 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:41.524050 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:41.524500 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:41.524611 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:41.527184 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:41.527191 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:41.527207 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:41.527236 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:41.528978 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:41.529407 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:41.529413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:41.529418 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:41.529551 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:41.529841 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:41.529844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:41.529846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:41.530322 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:41.531008 (+ 686us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:41.531152 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 13:48:41.531584 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":125,"threads_started":1}
I20260926 13:48:41.534214 30068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:48:41.539278 30068 catalog_manager.cc:1347] Generated new cluster ID: d15436a1660145b3845501f38f019af2
I20260926 13:48:41.539327 30068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:48:41.562520 30068 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:48:41.563455 30068 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:48:41.567615 30068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56d5e9c47b914483a14f573caa792c32: Generated new TSK 0
I20260926 13:48:41.568256 30068 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.170.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
I20260926 13:48:41.600762 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:0
--local_ip_for_outbound_sockets=127.12.170.65
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:36577
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:45407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:48:41.703752 30077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:41.703989 30077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:41.704043 30077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:41.707963 30077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:41.708016 30077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:41.708092 30077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:48:41.713104 30077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:45407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:36577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:41.714010 30077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:41.714933 30077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:41.723464 30082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:41.723464 30085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:41.723558 30083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:41.723693 30077 server_base.cc:1034] running on GCE node
I20260926 13:48:41.724076 30077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:41.724663 30077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:41.725812 30077 hybrid_clock.cc:648] HybridClock initialized: now 1790430521725793 us; error 31 us; skew 500 ppm
Sep 26 13:48:41 dist-test-slave-hc1k krb5kdc[29970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430521, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:41.728345 30077 init.cc:377] Logged in from keytab as kudu/127.12.170.65@KRBTEST.COM (short username kudu)
I20260926 13:48:41.729260 30077 webserver.cc:460] Webserver started at http://127.12.170.65:38511/ using document root <none> and password file <none>
I20260926 13:48:41.729818 30077 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:41.729882 30077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:41.730118 30077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:41.732118 30077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-0/data/instance:
uuid: "1622663870ed45c6857123bf08d43eaa"
format_stamp: "Formatted at 2026-09-26 13:48:41 on dist-test-slave-hc1k"
server_key: "824bc7ff3672e7a729b97bf7d707bc7a"
server_key_iv: "500afa474fc0b543e25b2199ea84057a"
server_key_version: "encryptionkey@0"
I20260926 13:48:41.732656 30077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance:
uuid: "1622663870ed45c6857123bf08d43eaa"
format_stamp: "Formatted at 2026-09-26 13:48:41 on dist-test-slave-hc1k"
server_key: "824bc7ff3672e7a729b97bf7d707bc7a"
server_key_iv: "500afa474fc0b543e25b2199ea84057a"
server_key_version: "encryptionkey@0"
I20260926 13:48:41.736162 30077 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:48:41.738319 30092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:41.739485 30077 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:48:41.739578 30077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "1622663870ed45c6857123bf08d43eaa"
format_stamp: "Formatted at 2026-09-26 13:48:41 on dist-test-slave-hc1k"
server_key: "824bc7ff3672e7a729b97bf7d707bc7a"
server_key_iv: "500afa474fc0b543e25b2199ea84057a"
server_key_version: "encryptionkey@0"
I20260926 13:48:41.739684 30077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:41.749778 30077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:41.752259 30077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:41.752389 30077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:41.752830 30077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:48:41.753543 30077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:48:41.753587 30077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:41.753635 30077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:48:41.753661 30077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:41.761314 30077 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:40725
I20260926 13:48:41.761365 30205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:40725 every 8 connection(s)
I20260926 13:48:41.762068 30077 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:48:41.766162 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 30077
I20260926 13:48:41.766275 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance
I20260926 13:48:41.766525 12969 external_mini_cluster.cc:1428] Setting key a861edd51c58cd8d039351ddfd2d9650
Sep 26 13:48:41 dist-test-slave-hc1k krb5kdc[29970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430521, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.170.66@KRBTEST.COM; defaulting to no policy
I20260926 13:48:41.774778 30054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:41.763683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:52533 (local address 127.12.170.126:36577)
0926 13:48:41.763818 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:41.763821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:41.764536 (+ 715us) server_negotiation.cc:407] Connection header received
0926 13:48:41.765777 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:41.765781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:41.765836 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:41.765909 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:41.767081 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:41.767728 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:41.768792 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:41.769112 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:41.769225 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:41.771690 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:41.771694 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:41.771697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:41.771723 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:41.773010 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:41.773454 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:41.773457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:41.773459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:41.773529 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:41.773827 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:41.773830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:41.773832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:41.773992 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:41.774072 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:41.774511 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 13:48:41.774624 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 13:48:41.775503 30208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:41.763998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:36577 (local address 127.12.170.65:52533)
0926 13:48:41.764403 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:41.764443 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:41.765585 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:41.766091 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:41.766097 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:41.766443 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:48:41.766944 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:41.766964 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:41.767841 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:41.767845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:41.768672 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:41.768682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:41.769296 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:41.769301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:41.769485 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:41.769998 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:48:41.770018 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:48:41.771515 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:48:41.773110 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:41.773116 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:41.773129 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:41.773366 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:41.773605 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:41.773608 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:41.773610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:41.773745 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:41.774086 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:48:41.774092 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:48:41.774305 (+ 213us) client_negotiation.cc:769] Sending connection context
0926 13:48:41.774455 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 13:48:41.774748 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":110,"threads_started":1}
I20260926 13:48:41.776595 30206 heartbeater.cc:347] Connected to a master server at 127.12.170.126:36577
I20260926 13:48:41.776803 30206 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.66@KRBTEST.COM" created.
I20260926 13:48:41.777267 30206 heartbeater.cc:511] Master 127.12.170.126:36577 requested a full tablet report, sending...
I20260926 13:48:41.778849 30015 ts_manager.cc:194] Registered new tserver with Master: 1622663870ed45c6857123bf08d43eaa (127.12.170.65:40725)
I20260926 13:48:41.780084 30015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.65@KRBTEST.COM'} at 127.12.170.65:52533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
I20260926 13:48:41.792722 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:0
--local_ip_for_outbound_sockets=127.12.170.66
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:36577
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:45407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:48:41.897156 30211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:41.897392 30211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:41.897449 30211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:41.901367 30211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:41.901417 30211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:41.901492 30211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:48:41.906515 30211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:45407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:36577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:41.907460 30211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:41.908432 30211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:41.917280 30219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:41.917364 30216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:41.917300 30217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:41.917569 30211 server_base.cc:1034] running on GCE node
I20260926 13:48:41.917930 30211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:41.918506 30211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:41.919688 30211 hybrid_clock.cc:648] HybridClock initialized: now 1790430521919661 us; error 35 us; skew 500 ppm
Sep 26 13:48:41 dist-test-slave-hc1k krb5kdc[29970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430521, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:41.922503 30211 init.cc:377] Logged in from keytab as kudu/127.12.170.66@KRBTEST.COM (short username kudu)
I20260926 13:48:41.923359 30211 webserver.cc:460] Webserver started at http://127.12.170.66:40899/ using document root <none> and password file <none>
I20260926 13:48:41.923960 30211 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:41.924026 30211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:41.924213 30211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:41.926214 30211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-1/data/instance:
uuid: "32fca76c0b5d473eae4a67889c147603"
format_stamp: "Formatted at 2026-09-26 13:48:41 on dist-test-slave-hc1k"
server_key: "0fd1995fc23963c4e54dc83992a065b1"
server_key_iv: "b815e4cef71b07b72863222a557922fe"
server_key_version: "encryptionkey@0"
I20260926 13:48:41.926730 30211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance:
uuid: "32fca76c0b5d473eae4a67889c147603"
format_stamp: "Formatted at 2026-09-26 13:48:41 on dist-test-slave-hc1k"
server_key: "0fd1995fc23963c4e54dc83992a065b1"
server_key_iv: "b815e4cef71b07b72863222a557922fe"
server_key_version: "encryptionkey@0"
I20260926 13:48:41.930542 30211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 13:48:41.932914 30226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:41.934118 30211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:48:41.934228 30211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "32fca76c0b5d473eae4a67889c147603"
format_stamp: "Formatted at 2026-09-26 13:48:41 on dist-test-slave-hc1k"
server_key: "0fd1995fc23963c4e54dc83992a065b1"
server_key_iv: "b815e4cef71b07b72863222a557922fe"
server_key_version: "encryptionkey@0"
I20260926 13:48:41.934333 30211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:41.954860 30211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:41.957983 30211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:41.958148 30211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:41.958735 30211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:48:41.959544 30211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:48:41.959589 30211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:41.959640 30211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:48:41.959689 30211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:41.968012 30211 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:35253
I20260926 13:48:41.968065 30339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:35253 every 8 connection(s)
I20260926 13:48:41.968829 30211 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
Sep 26 13:48:41 dist-test-slave-hc1k krb5kdc[29970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430521, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:48:41.977792 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 30211
I20260926 13:48:41.977905 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance
I20260926 13:48:41.978206 12969 external_mini_cluster.cc:1428] Setting key 25fbb375e81349eecf67e213b88a4f9b
I20260926 13:48:41.981317 30054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:41.970495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:34751 (local address 127.12.170.126:36577)
0926 13:48:41.970622 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:41.970626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:41.971403 (+ 777us) server_negotiation.cc:407] Connection header received
0926 13:48:41.972665 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:41.972668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:41.972722 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:41.972793 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:41.973841 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:41.974357 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:41.975440 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:41.975680 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:41.975771 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:41.978001 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:41.978005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:41.978007 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:41.978029 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:41.979413 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:41.979904 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:41.979908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:41.979909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:41.980006 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:41.980321 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:41.980324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:41.980326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:41.980539 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:41.980639 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:41.981038 (+ 399us) server_negotiation.cc:299] Negotiation successful
0926 13:48:41.981163 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 13:48:41.982257 30342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:41.970848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:36577 (local address 127.12.170.66:34751)
0926 13:48:41.971274 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:41.971312 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:41.972465 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:41.972945 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:41.972952 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:41.973257 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:48:41.973707 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:41.973720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:41.974467 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:41.974470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:41.975324 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:41.975333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:41.975809 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:41.975813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:41.975975 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:41.976438 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:48:41.976453 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:48:41.977809 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:48:41.979520 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:41.979526 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:41.979539 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:41.979814 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:41.980090 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:41.980092 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:41.980093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:41.980242 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:41.980626 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:48:41.980630 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:48:41.980832 (+ 202us) client_negotiation.cc:769] Sending connection context
0926 13:48:41.980974 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 13:48:41.981325 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":115,"threads_started":1}
I20260926 13:48:41.983355 30340 heartbeater.cc:347] Connected to a master server at 127.12.170.126:36577
I20260926 13:48:41.983564 30340 heartbeater.cc:464] Registering TS with master...
I20260926 13:48:41.984064 30340 heartbeater.cc:511] Master 127.12.170.126:36577 requested a full tablet report, sending...
I20260926 13:48:41.985054 30015 ts_manager.cc:194] Registered new tserver with Master: 32fca76c0b5d473eae4a67889c147603 (127.12.170.66:35253)
I20260926 13:48:41.985571 30015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.66@KRBTEST.COM'} at 127.12.170.66:34751
WARNING: no policy specified for kudu/127.12.170.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
I20260926 13:48:42.002717 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:0
--local_ip_for_outbound_sockets=127.12.170.67
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:36577
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:45407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:48:42.104693 30345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:42.104905 30345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:42.104966 30345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:42.108870 30345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:42.108913 30345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:42.108969 30345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:48:42.113871 30345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:45407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:36577
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:42.114722 30345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:42.115567 30345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:48:42.124290 30345 server_base.cc:1034] running on GCE node
W20260926 13:48:42.124312 30350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:42.124311 30351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:42.124503 30353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:42.124864 30345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:42.125424 30345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:42.126577 30345 hybrid_clock.cc:648] HybridClock initialized: now 1790430522126562 us; error 27 us; skew 500 ppm
Sep 26 13:48:42 dist-test-slave-hc1k krb5kdc[29970](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430522, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:42.129079 30345 init.cc:377] Logged in from keytab as kudu/127.12.170.67@KRBTEST.COM (short username kudu)
I20260926 13:48:42.129873 30345 webserver.cc:460] Webserver started at http://127.12.170.67:38325/ using document root <none> and password file <none>
I20260926 13:48:42.130360 30345 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:42.130416 30345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:42.130559 30345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:42.132427 30345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-2/data/instance:
uuid: "6b43948075c0424896f1a6ff96035e6f"
format_stamp: "Formatted at 2026-09-26 13:48:42 on dist-test-slave-hc1k"
server_key: "949068566c4162b193f7d3f5a9bd5355"
server_key_iv: "9c3bf602bf3c4dc0b059f86a0bc05b57"
server_key_version: "encryptionkey@0"
I20260926 13:48:42.132893 30345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance:
uuid: "6b43948075c0424896f1a6ff96035e6f"
format_stamp: "Formatted at 2026-09-26 13:48:42 on dist-test-slave-hc1k"
server_key: "949068566c4162b193f7d3f5a9bd5355"
server_key_iv: "9c3bf602bf3c4dc0b059f86a0bc05b57"
server_key_version: "encryptionkey@0"
I20260926 13:48:42.136216 30345 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 13:48:42.138417 30360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:42.139525 30345 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:48:42.139619 30345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "6b43948075c0424896f1a6ff96035e6f"
format_stamp: "Formatted at 2026-09-26 13:48:42 on dist-test-slave-hc1k"
server_key: "949068566c4162b193f7d3f5a9bd5355"
server_key_iv: "9c3bf602bf3c4dc0b059f86a0bc05b57"
server_key_version: "encryptionkey@0"
I20260926 13:48:42.139737 30345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:42.176528 30345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:42.179379 30345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:42.179536 30345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:42.180079 30345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:48:42.180824 30345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:48:42.180868 30345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:42.180917 30345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:48:42.180943 30345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:42.189333 30345 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:40679
I20260926 13:48:42.189383 30473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:40679 every 8 connection(s)
I20260926 13:48:42.190128 30345 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:48:42.198050 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 30345
I20260926 13:48:42.198148 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance
I20260926 13:48:42.198407 12969 external_mini_cluster.cc:1428] Setting key beba427c466b489bb9ddf9df8397797f
Sep 26 13:48:42 dist-test-slave-hc1k krb5kdc[29970](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430522, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:48:42.204061 30054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:42.191801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:33781 (local address 127.12.170.126:36577)
0926 13:48:42.191920 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:42.191924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:42.193305 (+ 1381us) server_negotiation.cc:407] Connection header received
0926 13:48:42.194809 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:42.194812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:42.194870 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:42.194939 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:42.196370 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.197092 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.198454 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.198677 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.198784 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:42.201174 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:42.201177 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:42.201179 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:42.201203 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:42.202345 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:42.202810 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:42.202813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:42.202814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:42.202887 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:42.203172 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:42.203174 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:42.203175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:42.203333 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:42.203402 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:42.203852 (+ 450us) server_negotiation.cc:299] Negotiation successful
0926 13:48:42.203944 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 13:48:42.204833 30476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:42.192103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:36577 (local address 127.12.170.67:33781)
0926 13:48:42.193145 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:42.193193 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:42.194588 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:42.195117 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:42.195126 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:42.195600 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:48:42.196230 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:42.196254 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.197206 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.197210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:42.198349 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:42.198359 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.198808 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.198812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:42.199001 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:42.199535 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:48:42.199555 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:48:42.201026 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:48:42.202455 (+ 1429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:42.202465 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:42.202476 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:42.202718 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:42.202964 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:42.202967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:42.202968 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:42.203102 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:42.203401 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:48:42.203406 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:48:42.203612 (+ 206us) client_negotiation.cc:769] Sending connection context
0926 13:48:42.203802 (+ 190us) client_negotiation.cc:240] Negotiation successful
0926 13:48:42.204050 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"thread_start_us":124,"threads_started":1}
I20260926 13:48:42.205845 30474 heartbeater.cc:347] Connected to a master server at 127.12.170.126:36577
I20260926 13:48:42.206034 30474 heartbeater.cc:464] Registering TS with master...
I20260926 13:48:42.206450 30474 heartbeater.cc:511] Master 127.12.170.126:36577 requested a full tablet report, sending...
I20260926 13:48:42.207327 30015 ts_manager.cc:194] Registered new tserver with Master: 6b43948075c0424896f1a6ff96035e6f (127.12.170.67:40679)
I20260926 13:48:42.207894 30015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.67@KRBTEST.COM'} at 127.12.170.67:33781
I20260926 13:48:42.211838 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:48:42.220098 30054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:42.213568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55482 (local address 127.12.170.126:36577)
0926 13:48:42.213678 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:42.213683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:42.213844 (+ 161us) server_negotiation.cc:407] Connection header received
0926 13:48:42.213953 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:42.213956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:42.213995 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:42.214055 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:42.214707 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.215209 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.216050 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.216257 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.216338 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:42.217548 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:42.217551 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:42.217553 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:42.217574 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:42.218565 (+ 991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:42.218925 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:42.218929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:42.218931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:42.219004 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:42.219240 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:42.219244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:42.219245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:42.219461 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:42.219581 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:42.219855 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 13:48:42.219957 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 13:48:42.222460 30015 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55482:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:48:42.225337 30015 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:48:42.238911 30486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:42.233046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:35253 (local address 127.0.0.1:40872)
0926 13:48:42.233562 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:42.233573 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:42.233641 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:42.234880 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:42.234882 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:42.234899 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:42.234975 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:42.234980 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.236286 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.236290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:42.237448 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:42.237455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.238397 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.238400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:42.238515 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:42.238549 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 13:48:42.238648 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 13:48:42.238772 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":21,"thread_start_us":100,"threads_started":1}
I20260926 13:48:42.238960 30487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:42.233188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40872 (local address 127.12.170.66:35253)
0926 13:48:42.234329 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:42.234340 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:42.234363 (+ 23us) server_negotiation.cc:407] Connection header received
0926 13:48:42.234431 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:42.234435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:42.234612 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:42.234910 (+ 298us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:42.235056 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.236170 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.237560 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.238226 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.238396 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:42.238541 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:42.238760 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 13:48:42.238847 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1012,"thread_start_us":147,"threads_started":1}
I20260926 13:48:42.239344 30490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:42.234235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47932 (local address 127.12.170.67:40679)
0926 13:48:42.234637 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:42.234648 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:42.234685 (+ 37us) server_negotiation.cc:407] Connection header received
0926 13:48:42.234771 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:42.234776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:42.234958 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:42.235105 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:42.235451 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.236456 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.237719 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.238482 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.238566 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:42.238713 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:42.239135 (+ 422us) server_negotiation.cc:299] Negotiation successful
0926 13:48:42.239221 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":134,"threads_started":1}
I20260926 13:48:42.239370 30488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:42.233580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57946 (local address 127.12.170.65:40725)
0926 13:48:42.233944 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:42.233956 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:42.233979 (+ 23us) server_negotiation.cc:407] Connection header received
0926 13:48:42.234055 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:42.234060 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:42.234266 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:42.234415 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:42.234854 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.235896 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.237565 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.238351 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.238451 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:42.238580 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:42.238976 (+ 396us) server_negotiation.cc:299] Negotiation successful
0926 13:48:42.239258 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":135,"threads_started":1}
I20260926 13:48:42.239590 30485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:42.232931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:40725 (local address 127.0.0.1:57946)
0926 13:48:42.233378 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:42.233421 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:42.233534 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:42.234582 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:42.234588 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:42.234609 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:42.234756 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:42.234766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.236001 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.236005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:42.237340 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:42.237350 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.238640 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.238644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:42.238730 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:42.238753 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 13:48:42.238816 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 13:48:42.238882 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":25,"thread_start_us":59,"threads_started":1}
I20260926 13:48:42.239603 30489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:42.233964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:40679 (local address 127.0.0.1:47932)
0926 13:48:42.234535 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:42.234548 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:42.234630 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:42.235238 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:42.235241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:42.235257 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:42.235332 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:42.235337 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.236590 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.236594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:42.237603 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:42.237614 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.238764 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.238768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:42.238844 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:42.238863 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 13:48:42.238915 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 13:48:42.239501 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":3,"thread_start_us":89,"threads_started":1}
I20260926 13:48:42.241832 30140 tablet_service.cc:1467] Processing CreateTablet for tablet 812c784d5f004d3eb8c0d2d10d5db08a (DEFAULT_TABLE table=test-table [id=1bdfbc18b237475892bb6b9706859c1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:48:42.241840 30274 tablet_service.cc:1467] Processing CreateTablet for tablet 812c784d5f004d3eb8c0d2d10d5db08a (DEFAULT_TABLE table=test-table [id=1bdfbc18b237475892bb6b9706859c1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:48:42.242151 30408 tablet_service.cc:1467] Processing CreateTablet for tablet 812c784d5f004d3eb8c0d2d10d5db08a (DEFAULT_TABLE table=test-table [id=1bdfbc18b237475892bb6b9706859c1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:48:42.242795 30140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 812c784d5f004d3eb8c0d2d10d5db08a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:42.242858 30274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 812c784d5f004d3eb8c0d2d10d5db08a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:42.242990 30408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 812c784d5f004d3eb8c0d2d10d5db08a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:42.249153 30491 tablet_bootstrap.cc:492] T 812c784d5f004d3eb8c0d2d10d5db08a P 1622663870ed45c6857123bf08d43eaa: Bootstrap starting.
I20260926 13:48:42.249910 30492 tablet_bootstrap.cc:492] T 812c784d5f004d3eb8c0d2d10d5db08a P 6b43948075c0424896f1a6ff96035e6f: Bootstrap starting.
I20260926 13:48:42.249910 30493 tablet_bootstrap.cc:492] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603: Bootstrap starting.
I20260926 13:48:42.251124 30491 tablet_bootstrap.cc:654] T 812c784d5f004d3eb8c0d2d10d5db08a P 1622663870ed45c6857123bf08d43eaa: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:42.251950 30493 tablet_bootstrap.cc:654] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:42.251950 30492 tablet_bootstrap.cc:654] T 812c784d5f004d3eb8c0d2d10d5db08a P 6b43948075c0424896f1a6ff96035e6f: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:42.252137 30491 log.cc:824] T 812c784d5f004d3eb8c0d2d10d5db08a P 1622663870ed45c6857123bf08d43eaa: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:42.253109 30492 log.cc:824] T 812c784d5f004d3eb8c0d2d10d5db08a P 6b43948075c0424896f1a6ff96035e6f: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:42.253109 30493 log.cc:824] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:42.254086 30491 tablet_bootstrap.cc:492] T 812c784d5f004d3eb8c0d2d10d5db08a P 1622663870ed45c6857123bf08d43eaa: No bootstrap required, opened a new log
I20260926 13:48:42.254346 30491 ts_tablet_manager.cc:1397] T 812c784d5f004d3eb8c0d2d10d5db08a P 1622663870ed45c6857123bf08d43eaa: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260926 13:48:42.255146 30493 tablet_bootstrap.cc:492] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603: No bootstrap required, opened a new log
I20260926 13:48:42.255395 30493 ts_tablet_manager.cc:1397] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 13:48:42.255517 30492 tablet_bootstrap.cc:492] T 812c784d5f004d3eb8c0d2d10d5db08a P 6b43948075c0424896f1a6ff96035e6f: No bootstrap required, opened a new log
I20260926 13:48:42.255720 30492 ts_tablet_manager.cc:1397] T 812c784d5f004d3eb8c0d2d10d5db08a P 6b43948075c0424896f1a6ff96035e6f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 13:48:42.257282 30491 raft_consensus.cc:348] T 812c784d5f004d3eb8c0d2d10d5db08a P 1622663870ed45c6857123bf08d43eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622663870ed45c6857123bf08d43eaa" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40725 } } peers { permanent_uuid: "32fca76c0b5d473eae4a67889c147603" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35253 } } peers { permanent_uuid: "6b43948075c0424896f1a6ff96035e6f" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 40679 } }
I20260926 13:48:42.257475 30491 raft_consensus.cc:374] T 812c784d5f004d3eb8c0d2d10d5db08a P 1622663870ed45c6857123bf08d43eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:42.257525 30491 raft_consensus.cc:729] T 812c784d5f004d3eb8c0d2d10d5db08a P 1622663870ed45c6857123bf08d43eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1622663870ed45c6857123bf08d43eaa, State: Initialized, Role: FOLLOWER
I20260926 13:48:42.257899 30491 consensus_queue.cc:260] T 812c784d5f004d3eb8c0d2d10d5db08a P 1622663870ed45c6857123bf08d43eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622663870ed45c6857123bf08d43eaa" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40725 } } peers { permanent_uuid: "32fca76c0b5d473eae4a67889c147603" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35253 } } peers { permanent_uuid: "6b43948075c0424896f1a6ff96035e6f" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 40679 } }
I20260926 13:48:42.258463 30206 heartbeater.cc:503] Master 127.12.170.126:36577 was elected leader, sending a full tablet report...
I20260926 13:48:42.258746 30491 ts_tablet_manager.cc:1428] T 812c784d5f004d3eb8c0d2d10d5db08a P 1622663870ed45c6857123bf08d43eaa: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 13:48:42.259146 30492 raft_consensus.cc:348] T 812c784d5f004d3eb8c0d2d10d5db08a P 6b43948075c0424896f1a6ff96035e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622663870ed45c6857123bf08d43eaa" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40725 } } peers { permanent_uuid: "32fca76c0b5d473eae4a67889c147603" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35253 } } peers { permanent_uuid: "6b43948075c0424896f1a6ff96035e6f" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 40679 } }
I20260926 13:48:42.259377 30492 raft_consensus.cc:374] T 812c784d5f004d3eb8c0d2d10d5db08a P 6b43948075c0424896f1a6ff96035e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:42.259428 30492 raft_consensus.cc:729] T 812c784d5f004d3eb8c0d2d10d5db08a P 6b43948075c0424896f1a6ff96035e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b43948075c0424896f1a6ff96035e6f, State: Initialized, Role: FOLLOWER
I20260926 13:48:42.259881 30492 consensus_queue.cc:260] T 812c784d5f004d3eb8c0d2d10d5db08a P 6b43948075c0424896f1a6ff96035e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622663870ed45c6857123bf08d43eaa" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40725 } } peers { permanent_uuid: "32fca76c0b5d473eae4a67889c147603" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35253 } } peers { permanent_uuid: "6b43948075c0424896f1a6ff96035e6f" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 40679 } }
I20260926 13:48:42.260031 30493 raft_consensus.cc:348] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622663870ed45c6857123bf08d43eaa" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40725 } } peers { permanent_uuid: "32fca76c0b5d473eae4a67889c147603" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35253 } } peers { permanent_uuid: "6b43948075c0424896f1a6ff96035e6f" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 40679 } }
I20260926 13:48:42.260275 30493 raft_consensus.cc:374] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:42.260568 30474 heartbeater.cc:503] Master 127.12.170.126:36577 was elected leader, sending a full tablet report...
I20260926 13:48:42.260329 30493 raft_consensus.cc:729] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32fca76c0b5d473eae4a67889c147603, State: Initialized, Role: FOLLOWER
I20260926 13:48:42.260802 30492 ts_tablet_manager.cc:1428] T 812c784d5f004d3eb8c0d2d10d5db08a P 6b43948075c0424896f1a6ff96035e6f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 13:48:42.261335 30493 consensus_queue.cc:260] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622663870ed45c6857123bf08d43eaa" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40725 } } peers { permanent_uuid: "32fca76c0b5d473eae4a67889c147603" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35253 } } peers { permanent_uuid: "6b43948075c0424896f1a6ff96035e6f" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 40679 } }
I20260926 13:48:42.261911 30340 heartbeater.cc:503] Master 127.12.170.126:36577 was elected leader, sending a full tablet report...
I20260926 13:48:42.262753 30493 ts_tablet_manager.cc:1428] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260926 13:48:42.263012 30207 tablet.cc:2369] T 812c784d5f004d3eb8c0d2d10d5db08a P 1622663870ed45c6857123bf08d43eaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:48:42.366384 30499 raft_consensus.cc:482] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:48:42.366570 30499 raft_consensus.cc:504] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622663870ed45c6857123bf08d43eaa" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40725 } } peers { permanent_uuid: "32fca76c0b5d473eae4a67889c147603" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35253 } } peers { permanent_uuid: "6b43948075c0424896f1a6ff96035e6f" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 40679 } }
I20260926 13:48:42.367687 30499 leader_election.cc:288] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1622663870ed45c6857123bf08d43eaa (127.12.170.65:40725), 6b43948075c0424896f1a6ff96035e6f (127.12.170.67:40679)
I20260926 13:48:42.371225 30342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:42.367925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:40679 (local address 127.12.170.66:57015)
0926 13:48:42.368124 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:42.368139 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:42.368247 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:42.368498 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:42.368501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:42.368519 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:42.368643 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:42.368650 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.369380 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.369383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:42.370161 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:42.370171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.370798 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.370802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:42.370909 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:42.370928 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 13:48:42.371002 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 13:48:42.371076 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":1}
I20260926 13:48:42.371225 30490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:42.368030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:57015 (local address 127.12.170.67:40679)
0926 13:48:42.368147 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:42.368152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:42.368231 (+ 79us) server_negotiation.cc:407] Connection header received
0926 13:48:42.368359 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:42.368361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:42.368409 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:42.368479 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:42.368757 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.369270 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.370291 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.370684 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.370797 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:42.370855 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:42.371023 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 13:48:42.371088 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 13:48:42.371251 30488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:42.368024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:39749 (local address 127.12.170.65:40725)
0926 13:48:42.368144 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:42.368148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:42.368429 (+ 281us) server_negotiation.cc:407] Connection header received
0926 13:48:42.368559 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:42.368562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:42.368606 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:42.368668 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:42.368939 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.369422 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.370390 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.370765 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.370866 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:42.370915 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:42.371070 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 13:48:42.371140 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 13:48:42.371261 30500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:42.367925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:40725 (local address 127.12.170.66:39749)
0926 13:48:42.368324 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:42.368335 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:42.368438 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:42.368709 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:42.368711 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:42.368727 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:42.368840 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:42.368845 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.369524 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.369527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:42.370275 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:42.370283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.370869 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.370873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:42.370965 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:42.370984 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 13:48:42.371045 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 13:48:42.371108 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":41,"thread_start_us":101,"threads_started":1}
I20260926 13:48:42.371910 30428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812c784d5f004d3eb8c0d2d10d5db08a" candidate_uuid: "32fca76c0b5d473eae4a67889c147603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b43948075c0424896f1a6ff96035e6f" is_pre_election: true
I20260926 13:48:42.372011 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812c784d5f004d3eb8c0d2d10d5db08a" candidate_uuid: "32fca76c0b5d473eae4a67889c147603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1622663870ed45c6857123bf08d43eaa" is_pre_election: true
I20260926 13:48:42.372181 30428 raft_consensus.cc:2454] T 812c784d5f004d3eb8c0d2d10d5db08a P 6b43948075c0424896f1a6ff96035e6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32fca76c0b5d473eae4a67889c147603 in term 0.
I20260926 13:48:42.372336 30160 raft_consensus.cc:2454] T 812c784d5f004d3eb8c0d2d10d5db08a P 1622663870ed45c6857123bf08d43eaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32fca76c0b5d473eae4a67889c147603 in term 0.
I20260926 13:48:42.372634 30227 leader_election.cc:302] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32fca76c0b5d473eae4a67889c147603, 6b43948075c0424896f1a6ff96035e6f; no voters:
I20260926 13:48:42.372905 30499 raft_consensus.cc:2781] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:48:42.372978 30499 raft_consensus.cc:482] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:48:42.373006 30499 raft_consensus.cc:3037] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:42.374010 30499 raft_consensus.cc:504] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622663870ed45c6857123bf08d43eaa" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40725 } } peers { permanent_uuid: "32fca76c0b5d473eae4a67889c147603" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35253 } } peers { permanent_uuid: "6b43948075c0424896f1a6ff96035e6f" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 40679 } }
I20260926 13:48:42.374311 30499 leader_election.cc:288] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603 [CANDIDATE]: Term 1 election: Requested vote from peers 1622663870ed45c6857123bf08d43eaa (127.12.170.65:40725), 6b43948075c0424896f1a6ff96035e6f (127.12.170.67:40679)
I20260926 13:48:42.374626 30160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812c784d5f004d3eb8c0d2d10d5db08a" candidate_uuid: "32fca76c0b5d473eae4a67889c147603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1622663870ed45c6857123bf08d43eaa"
I20260926 13:48:42.374626 30428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812c784d5f004d3eb8c0d2d10d5db08a" candidate_uuid: "32fca76c0b5d473eae4a67889c147603" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b43948075c0424896f1a6ff96035e6f"
I20260926 13:48:42.374751 30160 raft_consensus.cc:3037] T 812c784d5f004d3eb8c0d2d10d5db08a P 1622663870ed45c6857123bf08d43eaa [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:42.374786 30428 raft_consensus.cc:3037] T 812c784d5f004d3eb8c0d2d10d5db08a P 6b43948075c0424896f1a6ff96035e6f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:42.376068 30160 raft_consensus.cc:2454] T 812c784d5f004d3eb8c0d2d10d5db08a P 1622663870ed45c6857123bf08d43eaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32fca76c0b5d473eae4a67889c147603 in term 1.
I20260926 13:48:42.376068 30428 raft_consensus.cc:2454] T 812c784d5f004d3eb8c0d2d10d5db08a P 6b43948075c0424896f1a6ff96035e6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32fca76c0b5d473eae4a67889c147603 in term 1.
I20260926 13:48:42.376363 30227 leader_election.cc:302] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32fca76c0b5d473eae4a67889c147603, 6b43948075c0424896f1a6ff96035e6f; no voters:
I20260926 13:48:42.376523 30499 raft_consensus.cc:2781] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:48:42.376716 30499 raft_consensus.cc:686] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603 [term 1 LEADER]: Becoming Leader. State: Replica: 32fca76c0b5d473eae4a67889c147603, State: Running, Role: LEADER
I20260926 13:48:42.377002 30499 consensus_queue.cc:237] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622663870ed45c6857123bf08d43eaa" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40725 } } peers { permanent_uuid: "32fca76c0b5d473eae4a67889c147603" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35253 } } peers { permanent_uuid: "6b43948075c0424896f1a6ff96035e6f" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 40679 } }
I20260926 13:48:42.379912 30014 catalog_manager.cc:5495] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603 reported cstate change: term changed from 0 to 1, leader changed from <none> to 32fca76c0b5d473eae4a67889c147603 (127.12.170.66). New cstate: current_term: 1 leader_uuid: "32fca76c0b5d473eae4a67889c147603" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1622663870ed45c6857123bf08d43eaa" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32fca76c0b5d473eae4a67889c147603" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b43948075c0424896f1a6ff96035e6f" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 40679 } health_report { overall_health: UNKNOWN } } }
I20260926 13:48:42.423872 30487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:42.420696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40874 (local address 127.12.170.66:35253)
0926 13:48:42.420840 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:42.420845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:42.420884 (+ 39us) server_negotiation.cc:407] Connection header received
0926 13:48:42.420957 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:42.420959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:42.421015 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:42.421090 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:48:42.421293 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.421773 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.422567 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.422765 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.422827 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:42.423071 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:48:42.423463 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:48:42.423545 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:42.423638 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 13:48:42.423710 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:48:42.429431 30428 raft_consensus.cc:1261] T 812c784d5f004d3eb8c0d2d10d5db08a P 6b43948075c0424896f1a6ff96035e6f [term 1 FOLLOWER]: Refusing update from remote peer 32fca76c0b5d473eae4a67889c147603: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:48:42.429430 30160 raft_consensus.cc:1261] T 812c784d5f004d3eb8c0d2d10d5db08a P 1622663870ed45c6857123bf08d43eaa [term 1 FOLLOWER]: Refusing update from remote peer 32fca76c0b5d473eae4a67889c147603: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:48:42.430047 30501 consensus_queue.cc:1035] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b43948075c0424896f1a6ff96035e6f" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 40679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:48:42.430253 30499 consensus_queue.cc:1035] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1622663870ed45c6857123bf08d43eaa" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:48:42.437510 30505 mvcc.cc:205] Tried to move back new op lower bound from 7333603419863748608 to 7333603419658375168. Current Snapshot: MvccSnapshot[applied={T|T < 7333603419863748608}]
I20260926 13:48:42.438773 30506 mvcc.cc:205] Tried to move back new op lower bound from 7333603419863748608 to 7333603419658375168. Current Snapshot: MvccSnapshot[applied={T|T < 7333603419863748608}]
I20260926 13:48:42.439742 30504 mvcc.cc:205] Tried to move back new op lower bound from 7333603419863748608 to 7333603419658375168. Current Snapshot: MvccSnapshot[applied={T|T < 7333603419863748608 or (T in {7333603419863748608})}]
I20260926 13:48:42.444542 30014 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:48:42.444669 30014 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55482:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:48:42.446699 30014 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56d5e9c47b914483a14f573caa792c32: Sending DeleteTablet for 3 replicas of tablet 812c784d5f004d3eb8c0d2d10d5db08a
I20260926 13:48:42.447355 30140 tablet_service.cc:1514] Processing DeleteTablet for tablet 812c784d5f004d3eb8c0d2d10d5db08a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:48:42 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:57946
I20260926 13:48:42.447335 30274 tablet_service.cc:1514] Processing DeleteTablet for tablet 812c784d5f004d3eb8c0d2d10d5db08a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:48:42 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:40872
I20260926 13:48:42.447788 30514 tablet_replica.cc:330] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603: stopping tablet replica
I20260926 13:48:42.447795 30513 tablet_replica.cc:330] T 812c784d5f004d3eb8c0d2d10d5db08a P 1622663870ed45c6857123bf08d43eaa: stopping tablet replica
I20260926 13:48:42.447799 30408 tablet_service.cc:1514] Processing DeleteTablet for tablet 812c784d5f004d3eb8c0d2d10d5db08a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:48:42 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:47932
I20260926 13:48:42.448056 30513 raft_consensus.cc:2229] T 812c784d5f004d3eb8c0d2d10d5db08a P 1622663870ed45c6857123bf08d43eaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:48:42.448179 30514 raft_consensus.cc:2229] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:48:42.448341 30513 raft_consensus.cc:2258] T 812c784d5f004d3eb8c0d2d10d5db08a P 1622663870ed45c6857123bf08d43eaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:48:42.448629 30515 tablet_replica.cc:330] T 812c784d5f004d3eb8c0d2d10d5db08a P 6b43948075c0424896f1a6ff96035e6f: stopping tablet replica
I20260926 13:48:42.448865 30515 raft_consensus.cc:2229] T 812c784d5f004d3eb8c0d2d10d5db08a P 6b43948075c0424896f1a6ff96035e6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:48:42.448999 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 30077
I20260926 13:48:42.449577 30515 raft_consensus.cc:2258] T 812c784d5f004d3eb8c0d2d10d5db08a P 6b43948075c0424896f1a6ff96035e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:48:42.450842 30515 ts_tablet_manager.cc:1905] T 812c784d5f004d3eb8c0d2d10d5db08a P 6b43948075c0424896f1a6ff96035e6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:48:42.452349 30514 raft_consensus.cc:2258] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:48:42.453161 30515 ts_tablet_manager.cc:1918] T 812c784d5f004d3eb8c0d2d10d5db08a P 6b43948075c0424896f1a6ff96035e6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:48:42.453251 30515 log.cc:1196] T 812c784d5f004d3eb8c0d2d10d5db08a P 6b43948075c0424896f1a6ff96035e6f: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/812c784d5f004d3eb8c0d2d10d5db08a
I20260926 13:48:42.453552 30515 ts_tablet_manager.cc:1939] T 812c784d5f004d3eb8c0d2d10d5db08a P 6b43948075c0424896f1a6ff96035e6f: Deleting consensus metadata
I20260926 13:48:42.453853 30514 ts_tablet_manager.cc:1905] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:48:42.454772 29999 catalog_manager.cc:4841] TS 6b43948075c0424896f1a6ff96035e6f (127.12.170.67:40679): tablet 812c784d5f004d3eb8c0d2d10d5db08a (table test-table [id=1bdfbc18b237475892bb6b9706859c1b]) successfully deleted
W20260926 13:48:42.455818 30002 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.170.65:40725 (error 108)
I20260926 13:48:42.456278 30514 ts_tablet_manager.cc:1918] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:48:42.456360 30514 log.cc:1196] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790430291939677-12969-0/minicluster-data/ts-1/wal/wals/812c784d5f004d3eb8c0d2d10d5db08a
I20260926 13:48:42.456590 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 30211
I20260926 13:48:42.456627 30514 ts_tablet_manager.cc:1939] T 812c784d5f004d3eb8c0d2d10d5db08a P 32fca76c0b5d473eae4a67889c147603: Deleting consensus metadata
I20260926 13:48:42.456744 30485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:42.456469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:40725 (local address 127.0.0.1:57948)
0926 13:48:42.456588 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:42.456657 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.65:40725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 13:48:42.457460 30002 catalog_manager.cc:4584] TS 1622663870ed45c6857123bf08d43eaa (127.12.170.65:40725): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 812c784d5f004d3eb8c0d2d10d5db08a: Network error: Client connection negotiation failed: client connection to 127.12.170.65:40725: connect: Connection refused (error 111)
I20260926 13:48:42.461630 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 30345
I20260926 13:48:42.463893 30485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:42.461713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:35253 (local address 127.0.0.1:40882)
0926 13:48:42.463788 (+ 2075us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:42.463835 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.66:35253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2002}
I20260926 13:48:42.465826 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 29983
2026-09-26T13:48:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2195 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T13:48:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:48:42Z Disabled control of system clock
I20260926 13:48:42.492137 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:39489
--webserver_interface=127.12.170.126
--webserver_port=0
--builtin_ntp_servers=127.12.170.84:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:39489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:48:42.593675 30520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:42.593918 30520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:42.593950 30520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:42.597795 30520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:48:42.597846 30520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:42.597860 30520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:42.597875 30520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:48:42.597888 30520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:48:42.602371 30520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:39489
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:39489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:42.603260 30520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:42.604503 30520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:42.612282 30525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:42.612340 30520 server_base.cc:1034] running on GCE node
W20260926 13:48:42.612323 30528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:42.612298 30526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:42.613051 30520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:42.614018 30520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:42.615170 30520 hybrid_clock.cc:648] HybridClock initialized: now 1790430522615151 us; error 33 us; skew 500 ppm
I20260926 13:48:42.619486 30520 webserver.cc:460] Webserver started at http://127.12.170.126:33627/ using document root <none> and password file <none>
I20260926 13:48:42.620051 30520 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:42.620116 30520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:42.620302 30520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:42.622179 30520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "449fa7c144a84151b0b4b401ba33216e"
format_stamp: "Formatted at 2026-09-26 13:48:42 on dist-test-slave-hc1k"
server_key: "deb74cdf6e8857d270fb0979484c39d8"
server_key_iv: "a01ab50393f6e32a96dbda664eed2db1"
server_key_version: "encryptionkey@0"
I20260926 13:48:42.622674 30520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "449fa7c144a84151b0b4b401ba33216e"
format_stamp: "Formatted at 2026-09-26 13:48:42 on dist-test-slave-hc1k"
server_key: "deb74cdf6e8857d270fb0979484c39d8"
server_key_iv: "a01ab50393f6e32a96dbda664eed2db1"
server_key_version: "encryptionkey@0"
I20260926 13:48:42.625995 30520 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.002s
I20260926 13:48:42.628150 30534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:42.629244 30520 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:48:42.629346 30520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "449fa7c144a84151b0b4b401ba33216e"
format_stamp: "Formatted at 2026-09-26 13:48:42 on dist-test-slave-hc1k"
server_key: "deb74cdf6e8857d270fb0979484c39d8"
server_key_iv: "a01ab50393f6e32a96dbda664eed2db1"
server_key_version: "encryptionkey@0"
I20260926 13:48:42.629421 30520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:42.639456 30520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:42.640009 30520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:42.640131 30520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:42.645910 30520 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:39489
I20260926 13:48:42.645923 30586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:39489 every 8 connection(s)
I20260926 13:48:42.646744 30520 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:48:42.649483 30587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:42.654364 30587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 449fa7c144a84151b0b4b401ba33216e: Bootstrap starting.
I20260926 13:48:42.656579 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 30520
I20260926 13:48:42.656642 30587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 449fa7c144a84151b0b4b401ba33216e: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:42.656698 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:48:42.656991 12969 external_mini_cluster.cc:1428] Setting key f49d66f544a27df85ad12353626613f2
I20260926 13:48:42.657437 30587 log.cc:824] T 00000000000000000000000000000000 P 449fa7c144a84151b0b4b401ba33216e: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:42.659219 30587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 449fa7c144a84151b0b4b401ba33216e: No bootstrap required, opened a new log
I20260926 13:48:42.662372 30587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 449fa7c144a84151b0b4b401ba33216e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449fa7c144a84151b0b4b401ba33216e" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 39489 } }
I20260926 13:48:42.662557 30587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 449fa7c144a84151b0b4b401ba33216e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:42.662590 30587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 449fa7c144a84151b0b4b401ba33216e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449fa7c144a84151b0b4b401ba33216e, State: Initialized, Role: FOLLOWER
I20260926 13:48:42.662983 30587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 449fa7c144a84151b0b4b401ba33216e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449fa7c144a84151b0b4b401ba33216e" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 39489 } }
I20260926 13:48:42.663126 30587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 449fa7c144a84151b0b4b401ba33216e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:48:42.663189 30587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 449fa7c144a84151b0b4b401ba33216e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:48:42.663296 30587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 449fa7c144a84151b0b4b401ba33216e [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:42.664306 30587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 449fa7c144a84151b0b4b401ba33216e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449fa7c144a84151b0b4b401ba33216e" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 39489 } }
I20260926 13:48:42.664630 30587 leader_election.cc:302] T 00000000000000000000000000000000 P 449fa7c144a84151b0b4b401ba33216e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 449fa7c144a84151b0b4b401ba33216e; no voters:
I20260926 13:48:42.664909 30587 leader_election.cc:288] T 00000000000000000000000000000000 P 449fa7c144a84151b0b4b401ba33216e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:48:42.665011 30592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 449fa7c144a84151b0b4b401ba33216e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:48:42.665205 30592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 449fa7c144a84151b0b4b401ba33216e [term 1 LEADER]: Becoming Leader. State: Replica: 449fa7c144a84151b0b4b401ba33216e, State: Running, Role: LEADER
I20260926 13:48:42.665423 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:42.657998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48158 (local address 127.12.170.126:39489)
0926 13:48:42.658566 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:42.658580 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:42.658613 (+ 33us) server_negotiation.cc:407] Connection header received
0926 13:48:42.659209 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:42.659225 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:42.659549 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:42.659887 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:42.660276 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.661173 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.662051 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.662495 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.662633 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:42.663461 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:42.663467 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:42.663481 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:48:42.663970 (+ 489us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:42.664041 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:48:42.664066 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:42.664164 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:42.664276 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 13:48:42.664530 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":108,"threads_started":1}
I20260926 13:48:42.665504 30592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 449fa7c144a84151b0b4b401ba33216e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449fa7c144a84151b0b4b401ba33216e" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 39489 } }
I20260926 13:48:42.666039 30587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 449fa7c144a84151b0b4b401ba33216e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:48:42.666488 30592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 449fa7c144a84151b0b4b401ba33216e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 449fa7c144a84151b0b4b401ba33216e. Latest consensus state: current_term: 1 leader_uuid: "449fa7c144a84151b0b4b401ba33216e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449fa7c144a84151b0b4b401ba33216e" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 39489 } } }
I20260926 13:48:42.666586 30592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 449fa7c144a84151b0b4b401ba33216e [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:42.667832 30594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 449fa7c144a84151b0b4b401ba33216e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "449fa7c144a84151b0b4b401ba33216e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449fa7c144a84151b0b4b401ba33216e" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 39489 } } }
I20260926 13:48:42.667922 30594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 449fa7c144a84151b0b4b401ba33216e [sys.catalog]: This master's current role is: LEADER
W20260926 13:48:42.670209 30607 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 449fa7c144a84151b0b4b401ba33216e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:48:42.670288 30607 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:48:42.670436 30596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:48:42.671213 30596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:48:42.675964 30596 catalog_manager.cc:1347] Generated new cluster ID: 05fe7b730a5c4244a4590cc8942a7581
I20260926 13:48:42.676026 30596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:48:42.696555 30596 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:48:42.697341 30596 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:48:42.703207 30596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 449fa7c144a84151b0b4b401ba33216e: Generated new TSK 0
I20260926 13:48:42.703814 30596 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:48:42.712172 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:0
--local_ip_for_outbound_sockets=127.12.170.65
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:39489
--builtin_ntp_servers=127.12.170.84:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:48:42.827759 30611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:42.827986 30611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:42.828037 30611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:42.831902 30611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:42.831944 30611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:42.832012 30611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:48:42.836709 30611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:39489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:42.837634 30611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:42.838883 30611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:42.847604 30619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:42.847610 30616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:42.847615 30617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:42.847644 30611 server_base.cc:1034] running on GCE node
I20260926 13:48:42.848145 30611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:42.848758 30611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:42.849915 30611 hybrid_clock.cc:648] HybridClock initialized: now 1790430522849899 us; error 25 us; skew 500 ppm
I20260926 13:48:42.851550 30611 webserver.cc:460] Webserver started at http://127.12.170.65:34127/ using document root <none> and password file <none>
I20260926 13:48:42.852077 30611 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:42.852138 30611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:42.852316 30611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:42.854182 30611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-0/data/instance:
uuid: "56099527e1d24e159dd7611641ade1a9"
format_stamp: "Formatted at 2026-09-26 13:48:42 on dist-test-slave-hc1k"
server_key: "dc24de9eafe8c4fc6376b5880bd09d8a"
server_key_iv: "b98a2c2696693f90752d6d7c831d4349"
server_key_version: "encryptionkey@0"
I20260926 13:48:42.854687 30611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance:
uuid: "56099527e1d24e159dd7611641ade1a9"
format_stamp: "Formatted at 2026-09-26 13:48:42 on dist-test-slave-hc1k"
server_key: "dc24de9eafe8c4fc6376b5880bd09d8a"
server_key_iv: "b98a2c2696693f90752d6d7c831d4349"
server_key_version: "encryptionkey@0"
I20260926 13:48:42.857954 30611 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20260926 13:48:42.860131 30625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:42.861218 30611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:48:42.861326 30611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "56099527e1d24e159dd7611641ade1a9"
format_stamp: "Formatted at 2026-09-26 13:48:42 on dist-test-slave-hc1k"
server_key: "dc24de9eafe8c4fc6376b5880bd09d8a"
server_key_iv: "b98a2c2696693f90752d6d7c831d4349"
server_key_version: "encryptionkey@0"
I20260926 13:48:42.861410 30611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:42.887887 30611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:42.888513 30611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:42.888669 30611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:42.889185 30611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:48:42.890041 30611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:48:42.890087 30611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:42.890137 30611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:48:42.890161 30611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:42.898366 30611 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:39051
I20260926 13:48:42.898412 30738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:39051 every 8 connection(s)
I20260926 13:48:42.899161 30611 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:48:42.907406 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:42.900815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:51623 (local address 127.12.170.126:39489)
0926 13:48:42.900937 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:42.900941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:42.901876 (+ 935us) server_negotiation.cc:407] Connection header received
0926 13:48:42.903135 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:42.903137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:42.903184 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:42.903255 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:42.904182 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.904716 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.905838 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.906045 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.906110 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:42.907042 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:42.907045 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:42.907047 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:48:42.907059 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:42.907076 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:48:42.907084 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:42.907136 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:42.907249 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 13:48:42.907299 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:48:42.908048 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 30611
I20260926 13:48:42.908171 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance
I20260926 13:48:42.908411 30741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:42.901141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:39489 (local address 127.12.170.65:51623)
0926 13:48:42.901742 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:42.901779 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:42.902955 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:42.903447 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:42.903454 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:42.903537 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:48:42.904043 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:42.904057 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.904832 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.904835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:42.905740 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:42.905750 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:42.906153 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:42.906155 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:42.906292 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:42.906807 (+ 515us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:48:42.906809 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:48:42.906847 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:48:42.906908 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:48:42.907145 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:48:42.907150 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:48:42.907153 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:48:42.907246 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 13:48:42.907455 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":94,"threads_started":1}
I20260926 13:48:42.908516 12969 external_mini_cluster.cc:1428] Setting key f60ef4b485c2eed6495c9fa221fab7a0
I20260926 13:48:42.909581 30739 heartbeater.cc:347] Connected to a master server at 127.12.170.126:39489
I20260926 13:48:42.909802 30739 heartbeater.cc:464] Registering TS with master...
I20260926 13:48:42.910378 30739 heartbeater.cc:511] Master 127.12.170.126:39489 requested a full tablet report, sending...
I20260926 13:48:42.910601 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:0
--local_ip_for_outbound_sockets=127.12.170.66
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:39489
--builtin_ntp_servers=127.12.170.84:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 13:48:42.911993 30550 ts_manager.cc:194] Registered new tserver with Master: 56099527e1d24e159dd7611641ade1a9 (127.12.170.65:39051)
I20260926 13:48:42.913143 30550 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.170.65:51623
W20260926 13:48:43.013908 30742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:43.014133 30742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:43.014168 30742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:43.018132 30742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:43.018177 30742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:43.018230 30742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:48:43.022753 30742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:39489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:43.023689 30742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:43.024953 30742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:48:43.033722 30742 server_base.cc:1034] running on GCE node
W20260926 13:48:43.033660 30748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:43.033744 30750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:43.033891 30747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:43.034242 30742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:43.034802 30742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:43.035957 30742 hybrid_clock.cc:648] HybridClock initialized: now 1790430523035938 us; error 27 us; skew 500 ppm
I20260926 13:48:43.037642 30742 webserver.cc:460] Webserver started at http://127.12.170.66:46005/ using document root <none> and password file <none>
I20260926 13:48:43.038153 30742 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:43.038214 30742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:43.038389 30742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:43.040421 30742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-1/data/instance:
uuid: "28382a9f1b774d13be0e43755ab647bc"
format_stamp: "Formatted at 2026-09-26 13:48:43 on dist-test-slave-hc1k"
server_key: "650c26af4a2d020c298a54f593beae00"
server_key_iv: "5d07e50f81b04a161c5bcd62efeca291"
server_key_version: "encryptionkey@0"
I20260926 13:48:43.040930 30742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance:
uuid: "28382a9f1b774d13be0e43755ab647bc"
format_stamp: "Formatted at 2026-09-26 13:48:43 on dist-test-slave-hc1k"
server_key: "650c26af4a2d020c298a54f593beae00"
server_key_iv: "5d07e50f81b04a161c5bcd62efeca291"
server_key_version: "encryptionkey@0"
I20260926 13:48:43.044271 30742 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:48:43.046501 30756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:43.047569 30742 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:48:43.047693 30742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "28382a9f1b774d13be0e43755ab647bc"
format_stamp: "Formatted at 2026-09-26 13:48:43 on dist-test-slave-hc1k"
server_key: "650c26af4a2d020c298a54f593beae00"
server_key_iv: "5d07e50f81b04a161c5bcd62efeca291"
server_key_version: "encryptionkey@0"
I20260926 13:48:43.047784 30742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:43.061275 30742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:43.061821 30742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:43.061966 30742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:43.062448 30742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:48:43.063272 30742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:48:43.063316 30742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:43.063364 30742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:48:43.063392 30742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:43.071852 30742 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:35933
I20260926 13:48:43.071882 30869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:35933 every 8 connection(s)
I20260926 13:48:43.072618 30742 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:48:43.076218 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 30742
I20260926 13:48:43.076356 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance
I20260926 13:48:43.076678 12969 external_mini_cluster.cc:1428] Setting key 4f260c856007282603a07edfb994842a
I20260926 13:48:43.078703 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:0
--local_ip_for_outbound_sockets=127.12.170.67
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:39489
--builtin_ntp_servers=127.12.170.84:37447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 13:48:43.081876 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:43.074234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:41887 (local address 127.12.170.126:39489)
0926 13:48:43.074360 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:43.074364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:43.075220 (+ 856us) server_negotiation.cc:407] Connection header received
0926 13:48:43.076785 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:43.076789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:43.076843 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:43.076910 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:43.077864 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.078457 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.080026 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.080344 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.080440 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:43.081458 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:43.081461 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:43.081464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:48:43.081477 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:43.081499 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:48:43.081508 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:43.081572 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:43.081681 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 13:48:43.081746 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 13:48:43.082619 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:43.074494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:39489 (local address 127.12.170.66:41887)
0926 13:48:43.075091 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:43.075129 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:43.076550 (+ 1421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:43.077079 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:43.077089 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:43.077171 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:48:43.077737 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:43.077756 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.078588 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.078592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:43.079902 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:43.079915 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.080493 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.080496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:43.080632 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:43.081209 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:48:43.081211 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:48:43.081255 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:48:43.081314 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:48:43.081576 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:48:43.081580 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:48:43.081584 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:48:43.081664 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 13:48:43.081871 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":101,"threads_started":1}
I20260926 13:48:43.083827 30870 heartbeater.cc:347] Connected to a master server at 127.12.170.126:39489
I20260926 13:48:43.084012 30870 heartbeater.cc:464] Registering TS with master...
I20260926 13:48:43.084466 30870 heartbeater.cc:511] Master 127.12.170.126:39489 requested a full tablet report, sending...
I20260926 13:48:43.085450 30550 ts_manager.cc:194] Registered new tserver with Master: 28382a9f1b774d13be0e43755ab647bc (127.12.170.66:35933)
I20260926 13:48:43.086172 30550 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.170.66:41887
W20260926 13:48:43.182554 30873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:43.182770 30873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:43.182803 30873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:43.186645 30873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:43.186688 30873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:43.186736 30873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:48:43.191171 30873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:37447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:39489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:43.192050 30873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:43.193313 30873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:43.201936 30881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:43.202064 30873 server_base.cc:1034] running on GCE node
W20260926 13:48:43.201974 30879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:43.202011 30878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:43.202548 30873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:43.203140 30873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:43.204296 30873 hybrid_clock.cc:648] HybridClock initialized: now 1790430523204278 us; error 46 us; skew 500 ppm
I20260926 13:48:43.205886 30873 webserver.cc:460] Webserver started at http://127.12.170.67:44779/ using document root <none> and password file <none>
I20260926 13:48:43.206390 30873 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:43.206451 30873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:43.206614 30873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:43.208490 30873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-2/data/instance:
uuid: "5140ef4e2fd34aaa969943c38aa693e5"
format_stamp: "Formatted at 2026-09-26 13:48:43 on dist-test-slave-hc1k"
server_key: "7387dd66451502609599082b1d439dcf"
server_key_iv: "d86579aa1600b8870781b2e7149a9dbc"
server_key_version: "encryptionkey@0"
I20260926 13:48:43.208981 30873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance:
uuid: "5140ef4e2fd34aaa969943c38aa693e5"
format_stamp: "Formatted at 2026-09-26 13:48:43 on dist-test-slave-hc1k"
server_key: "7387dd66451502609599082b1d439dcf"
server_key_iv: "d86579aa1600b8870781b2e7149a9dbc"
server_key_version: "encryptionkey@0"
I20260926 13:48:43.212286 30873 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:48:43.214581 30887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:43.215690 30873 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:48:43.215799 30873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "5140ef4e2fd34aaa969943c38aa693e5"
format_stamp: "Formatted at 2026-09-26 13:48:43 on dist-test-slave-hc1k"
server_key: "7387dd66451502609599082b1d439dcf"
server_key_iv: "d86579aa1600b8870781b2e7149a9dbc"
server_key_version: "encryptionkey@0"
I20260926 13:48:43.215883 30873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:43.228901 30873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:43.229439 30873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:43.229581 30873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:43.230079 30873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:48:43.230891 30873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:48:43.230934 30873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:43.230981 30873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:48:43.231006 30873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:43.238896 30873 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:44849
I20260926 13:48:43.238924 31000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:44849 every 8 connection(s)
I20260926 13:48:43.239681 30873 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:48:43.244329 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 30873
I20260926 13:48:43.244424 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance
I20260926 13:48:43.244648 12969 external_mini_cluster.cc:1428] Setting key 59adf74c6f3f284abfb322013769b7e5
I20260926 13:48:43.248379 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:43.241434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:49771 (local address 127.12.170.126:39489)
0926 13:48:43.241571 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:43.241576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:43.242555 (+ 979us) server_negotiation.cc:407] Connection header received
0926 13:48:43.243840 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:43.243842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:43.243886 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:43.243956 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:43.244825 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.245437 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.246699 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.246925 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.247007 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:43.247948 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:43.247952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:43.247954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:48:43.247966 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:43.247988 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:48:43.247995 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:43.248051 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:43.248177 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 13:48:43.248235 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 13:48:43.249194 31003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:43.241822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:39489 (local address 127.12.170.67:49771)
0926 13:48:43.242420 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:43.242457 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:43.243621 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:43.244092 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:43.244099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:43.244179 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:48:43.244685 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:43.244698 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.245551 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.245554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:43.246599 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:43.246608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.247047 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.247051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:43.247198 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:43.247717 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:48:43.247718 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:48:43.247765 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:48:43.247823 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:48:43.248059 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:48:43.248067 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:48:43.248073 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 13:48:43.248170 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 13:48:43.248419 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":96,"threads_started":1}
I20260926 13:48:43.250077 31001 heartbeater.cc:347] Connected to a master server at 127.12.170.126:39489
I20260926 13:48:43.250283 31001 heartbeater.cc:464] Registering TS with master...
I20260926 13:48:43.250718 31001 heartbeater.cc:511] Master 127.12.170.126:39489 requested a full tablet report, sending...
I20260926 13:48:43.251578 30550 ts_manager.cc:194] Registered new tserver with Master: 5140ef4e2fd34aaa969943c38aa693e5 (127.12.170.67:44849)
I20260926 13:48:43.252118 30550 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.170.67:49771
I20260926 13:48:43.258241 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:48:43.263038 30591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:43.259991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48170 (local address 127.12.170.126:39489)
0926 13:48:43.260113 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:43.260117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:43.260239 (+ 122us) server_negotiation.cc:407] Connection header received
0926 13:48:43.260353 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:43.260355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:43.260392 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:43.260442 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:43.260630 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.261111 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.261858 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.262046 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.262104 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:43.262713 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:43.262716 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:43.262717 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:48:43.262728 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:43.262744 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:48:43.262760 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:43.262810 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:43.262877 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 13:48:43.262917 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 13:48:43.265496 30550 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:48:43.268493 30550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:48:43.281358 31014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:43.276001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57830 (local address 127.12.170.67:44849)
0926 13:48:43.276457 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:43.276468 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:43.276795 (+ 327us) server_negotiation.cc:407] Connection header received
0926 13:48:43.276896 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:43.276902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:43.277078 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:43.277217 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:43.277663 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.278480 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.279890 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.280661 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.280774 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:43.280952 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:43.281107 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 13:48:43.281200 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":86,"threads_started":1}
I20260926 13:48:43.281368 31013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:43.275787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:39051 (local address 127.0.0.1:40274)
0926 13:48:43.276184 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:43.276225 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:43.276336 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:43.277350 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:43.277356 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:43.277379 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:43.277521 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:43.277528 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.278830 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.278834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:43.280012 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:43.280020 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.280896 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.280899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:43.280992 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:43.281009 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 13:48:43.281057 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 13:48:43.281188 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":51,"spinlock_wait_cycles":7424,"thread_start_us":75,"threads_started":1}
I20260926 13:48:43.281374 31015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:43.275848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:44849 (local address 127.0.0.1:57830)
0926 13:48:43.276695 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:43.276705 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:43.276796 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:43.277396 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:43.277399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:43.277430 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:43.277536 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:43.277543 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.278583 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.278586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:43.279582 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:43.279592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.280781 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.280785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:43.280924 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:43.280968 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 13:48:43.281070 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 13:48:43.281188 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"mutex_wait_us":91,"spinlock_wait_cycles":10368,"thread_start_us":97,"threads_started":1}
I20260926 13:48:43.282188 31016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:43.276518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40274 (local address 127.12.170.65:39051)
0926 13:48:43.276873 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:43.276884 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:43.276905 (+ 21us) server_negotiation.cc:407] Connection header received
0926 13:48:43.276977 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:43.276981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:43.277166 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:43.277653 (+ 487us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:43.277781 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.278688 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.280132 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.280763 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.281756 (+ 993us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:43.281907 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:43.281954 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 13:48:43.282040 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":89,"threads_started":1}
I20260926 13:48:43.282312 31012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:43.276478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:35933 (local address 127.0.0.1:48966)
0926 13:48:43.277290 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:43.277303 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:43.277418 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:43.278065 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:43.278068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:43.278083 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:43.278155 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:43.278159 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.279705 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.279708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:43.280788 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:43.280798 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.281957 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.281960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:43.282042 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:43.282064 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 13:48:43.282140 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 13:48:43.282199 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"thread_start_us":85,"threads_started":1}
I20260926 13:48:43.282420 31017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:43.276693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48966 (local address 127.12.170.66:35933)
0926 13:48:43.277166 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:43.277177 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:43.277363 (+ 186us) server_negotiation.cc:407] Connection header received
0926 13:48:43.277547 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:43.277552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:43.277798 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:43.277907 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:43.278278 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.279557 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.280909 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.281845 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.281942 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:43.282093 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:43.282175 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 13:48:43.282270 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":96,"threads_started":1}
I20260926 13:48:43.284346 30935 tablet_service.cc:1467] Processing CreateTablet for tablet ec2ed0dcdd5a4c1ea5631fb3eda01e20 (DEFAULT_TABLE table=test-table [id=3cc6851d10d24d0fbb287632f69e8acc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:48:43.284767 30673 tablet_service.cc:1467] Processing CreateTablet for tablet ec2ed0dcdd5a4c1ea5631fb3eda01e20 (DEFAULT_TABLE table=test-table [id=3cc6851d10d24d0fbb287632f69e8acc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:48:43.284857 30804 tablet_service.cc:1467] Processing CreateTablet for tablet ec2ed0dcdd5a4c1ea5631fb3eda01e20 (DEFAULT_TABLE table=test-table [id=3cc6851d10d24d0fbb287632f69e8acc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:48:43.285437 30935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec2ed0dcdd5a4c1ea5631fb3eda01e20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:43.285655 30673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec2ed0dcdd5a4c1ea5631fb3eda01e20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:43.285782 30804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec2ed0dcdd5a4c1ea5631fb3eda01e20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:43.292098 31019 tablet_bootstrap.cc:492] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc: Bootstrap starting.
I20260926 13:48:43.291182 31018 tablet_bootstrap.cc:492] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 56099527e1d24e159dd7611641ade1a9: Bootstrap starting.
I20260926 13:48:43.292886 31020 tablet_bootstrap.cc:492] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 5140ef4e2fd34aaa969943c38aa693e5: Bootstrap starting.
I20260926 13:48:43.294329 31018 tablet_bootstrap.cc:654] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 56099527e1d24e159dd7611641ade1a9: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:43.294329 31019 tablet_bootstrap.cc:654] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:43.294445 31020 tablet_bootstrap.cc:654] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 5140ef4e2fd34aaa969943c38aa693e5: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:43.295073 31020 log.cc:824] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 5140ef4e2fd34aaa969943c38aa693e5: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:43.295142 31019 log.cc:824] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:43.295142 31018 log.cc:824] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 56099527e1d24e159dd7611641ade1a9: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:43.296782 31020 tablet_bootstrap.cc:492] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 5140ef4e2fd34aaa969943c38aa693e5: No bootstrap required, opened a new log
I20260926 13:48:43.296957 31020 ts_tablet_manager.cc:1397] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 5140ef4e2fd34aaa969943c38aa693e5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 13:48:43.296962 31018 tablet_bootstrap.cc:492] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 56099527e1d24e159dd7611641ade1a9: No bootstrap required, opened a new log
I20260926 13:48:43.296962 31019 tablet_bootstrap.cc:492] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc: No bootstrap required, opened a new log
I20260926 13:48:43.297122 31019 ts_tablet_manager.cc:1397] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 13:48:43.297122 31018 ts_tablet_manager.cc:1397] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 56099527e1d24e159dd7611641ade1a9: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260926 13:48:43.300168 31020 raft_consensus.cc:348] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 5140ef4e2fd34aaa969943c38aa693e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56099527e1d24e159dd7611641ade1a9" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39051 } } peers { permanent_uuid: "5140ef4e2fd34aaa969943c38aa693e5" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44849 } } peers { permanent_uuid: "28382a9f1b774d13be0e43755ab647bc" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35933 } }
I20260926 13:48:43.300379 31020 raft_consensus.cc:374] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 5140ef4e2fd34aaa969943c38aa693e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:43.300431 31020 raft_consensus.cc:729] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 5140ef4e2fd34aaa969943c38aa693e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5140ef4e2fd34aaa969943c38aa693e5, State: Initialized, Role: FOLLOWER
I20260926 13:48:43.300801 31020 consensus_queue.cc:260] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 5140ef4e2fd34aaa969943c38aa693e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56099527e1d24e159dd7611641ade1a9" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39051 } } peers { permanent_uuid: "5140ef4e2fd34aaa969943c38aa693e5" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44849 } } peers { permanent_uuid: "28382a9f1b774d13be0e43755ab647bc" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35933 } }
I20260926 13:48:43.301097 31019 raft_consensus.cc:348] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56099527e1d24e159dd7611641ade1a9" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39051 } } peers { permanent_uuid: "5140ef4e2fd34aaa969943c38aa693e5" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44849 } } peers { permanent_uuid: "28382a9f1b774d13be0e43755ab647bc" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35933 } }
I20260926 13:48:43.301096 31018 raft_consensus.cc:348] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 56099527e1d24e159dd7611641ade1a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56099527e1d24e159dd7611641ade1a9" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39051 } } peers { permanent_uuid: "5140ef4e2fd34aaa969943c38aa693e5" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44849 } } peers { permanent_uuid: "28382a9f1b774d13be0e43755ab647bc" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35933 } }
I20260926 13:48:43.301333 31019 raft_consensus.cc:374] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:43.301333 31018 raft_consensus.cc:374] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 56099527e1d24e159dd7611641ade1a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:43.301367 31001 heartbeater.cc:503] Master 127.12.170.126:39489 was elected leader, sending a full tablet report...
I20260926 13:48:43.301378 31018 raft_consensus.cc:729] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 56099527e1d24e159dd7611641ade1a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56099527e1d24e159dd7611641ade1a9, State: Initialized, Role: FOLLOWER
I20260926 13:48:43.301378 31019 raft_consensus.cc:729] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28382a9f1b774d13be0e43755ab647bc, State: Initialized, Role: FOLLOWER
I20260926 13:48:43.301713 31020 ts_tablet_manager.cc:1428] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 5140ef4e2fd34aaa969943c38aa693e5: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260926 13:48:43.301805 31018 consensus_queue.cc:260] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 56099527e1d24e159dd7611641ade1a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56099527e1d24e159dd7611641ade1a9" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39051 } } peers { permanent_uuid: "5140ef4e2fd34aaa969943c38aa693e5" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44849 } } peers { permanent_uuid: "28382a9f1b774d13be0e43755ab647bc" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35933 } }
I20260926 13:48:43.301800 31019 consensus_queue.cc:260] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56099527e1d24e159dd7611641ade1a9" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39051 } } peers { permanent_uuid: "5140ef4e2fd34aaa969943c38aa693e5" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44849 } } peers { permanent_uuid: "28382a9f1b774d13be0e43755ab647bc" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35933 } }
I20260926 13:48:43.302655 30870 heartbeater.cc:503] Master 127.12.170.126:39489 was elected leader, sending a full tablet report...
I20260926 13:48:43.302668 30739 heartbeater.cc:503] Master 127.12.170.126:39489 was elected leader, sending a full tablet report...
I20260926 13:48:43.302687 31019 ts_tablet_manager.cc:1428] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260926 13:48:43.302687 31018 ts_tablet_manager.cc:1428] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 56099527e1d24e159dd7611641ade1a9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260926 13:48:43.323415 30871 tablet.cc:2369] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:48:43.400246 30740 tablet.cc:2369] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 56099527e1d24e159dd7611641ade1a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:48:43.481722 31025 raft_consensus.cc:482] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:48:43.481971 31025 raft_consensus.cc:504] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56099527e1d24e159dd7611641ade1a9" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39051 } } peers { permanent_uuid: "5140ef4e2fd34aaa969943c38aa693e5" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44849 } } peers { permanent_uuid: "28382a9f1b774d13be0e43755ab647bc" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35933 } }
I20260926 13:48:43.483129 31025 leader_election.cc:288] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56099527e1d24e159dd7611641ade1a9 (127.12.170.65:39051), 5140ef4e2fd34aaa969943c38aa693e5 (127.12.170.67:44849)
I20260926 13:48:43.486922 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:43.483339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:39051 (local address 127.12.170.66:59935)
0926 13:48:43.483466 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:43.483481 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:43.483576 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:43.483944 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:43.483948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:43.483975 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:43.484118 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:43.484126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.484891 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.484894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:43.485865 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:43.485877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.486519 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.486523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:43.486619 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:43.486644 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 13:48:43.486719 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 13:48:43.486812 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":47}
I20260926 13:48:43.486930 31016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:43.483549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:59935 (local address 127.12.170.65:39051)
0926 13:48:43.483692 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:43.483696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:43.483710 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:48:43.483777 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:43.483780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:43.483829 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:43.483909 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:43.484239 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.484796 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.485982 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.486415 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.486501 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:43.486537 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:43.486732 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 13:48:43.486808 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 13:48:43.487125 31027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:43.483408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:44849 (local address 127.12.170.66:54167)
0926 13:48:43.483829 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:43.483842 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:43.483926 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:43.484182 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:43.484184 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:43.484200 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:43.484339 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:43.484346 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.485205 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.485208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:43.486043 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:43.486052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.486745 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.486749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:43.486843 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:43.486866 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 13:48:43.486936 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 13:48:43.486998 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":37,"thread_start_us":130,"threads_started":1}
I20260926 13:48:43.487140 31014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:43.483579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:54167 (local address 127.12.170.67:44849)
0926 13:48:43.483732 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:43.483735 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:43.483913 (+ 178us) server_negotiation.cc:407] Connection header received
0926 13:48:43.484035 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:43.484038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:43.484086 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:43.484208 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:43.484473 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.485105 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.486156 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.486625 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.486746 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:43.486787 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:43.486965 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 13:48:43.487034 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 13:48:43.487569 30693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec2ed0dcdd5a4c1ea5631fb3eda01e20" candidate_uuid: "28382a9f1b774d13be0e43755ab647bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56099527e1d24e159dd7611641ade1a9" is_pre_election: true
I20260926 13:48:43.487815 30955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec2ed0dcdd5a4c1ea5631fb3eda01e20" candidate_uuid: "28382a9f1b774d13be0e43755ab647bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5140ef4e2fd34aaa969943c38aa693e5" is_pre_election: true
I20260926 13:48:43.487882 30693 raft_consensus.cc:2454] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 56099527e1d24e159dd7611641ade1a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28382a9f1b774d13be0e43755ab647bc in term 0.
I20260926 13:48:43.488046 30955 raft_consensus.cc:2454] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 5140ef4e2fd34aaa969943c38aa693e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28382a9f1b774d13be0e43755ab647bc in term 0.
I20260926 13:48:43.488350 30759 leader_election.cc:302] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28382a9f1b774d13be0e43755ab647bc, 56099527e1d24e159dd7611641ade1a9; no voters:
I20260926 13:48:43.488612 31025 raft_consensus.cc:2781] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:48:43.488686 31025 raft_consensus.cc:482] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:48:43.488729 31025 raft_consensus.cc:3037] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:43.489681 31025 raft_consensus.cc:504] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56099527e1d24e159dd7611641ade1a9" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39051 } } peers { permanent_uuid: "5140ef4e2fd34aaa969943c38aa693e5" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44849 } } peers { permanent_uuid: "28382a9f1b774d13be0e43755ab647bc" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35933 } }
I20260926 13:48:43.489987 31025 leader_election.cc:288] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc [CANDIDATE]: Term 1 election: Requested vote from peers 56099527e1d24e159dd7611641ade1a9 (127.12.170.65:39051), 5140ef4e2fd34aaa969943c38aa693e5 (127.12.170.67:44849)
I20260926 13:48:43.490314 30955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec2ed0dcdd5a4c1ea5631fb3eda01e20" candidate_uuid: "28382a9f1b774d13be0e43755ab647bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5140ef4e2fd34aaa969943c38aa693e5"
I20260926 13:48:43.490350 30693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec2ed0dcdd5a4c1ea5631fb3eda01e20" candidate_uuid: "28382a9f1b774d13be0e43755ab647bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56099527e1d24e159dd7611641ade1a9"
I20260926 13:48:43.490434 30955 raft_consensus.cc:3037] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 5140ef4e2fd34aaa969943c38aa693e5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:43.490456 30693 raft_consensus.cc:3037] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 56099527e1d24e159dd7611641ade1a9 [term 0 FOLLOWER]: Advancing to term 1
W20260926 13:48:43.490517 31002 tablet.cc:2369] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 5140ef4e2fd34aaa969943c38aa693e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:48:43.491420 30693 raft_consensus.cc:2454] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 56099527e1d24e159dd7611641ade1a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28382a9f1b774d13be0e43755ab647bc in term 1.
I20260926 13:48:43.491514 30955 raft_consensus.cc:2454] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 5140ef4e2fd34aaa969943c38aa693e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28382a9f1b774d13be0e43755ab647bc in term 1.
I20260926 13:48:43.491706 30759 leader_election.cc:302] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28382a9f1b774d13be0e43755ab647bc, 56099527e1d24e159dd7611641ade1a9; no voters:
I20260926 13:48:43.491885 31025 raft_consensus.cc:2781] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:48:43.492084 31025 raft_consensus.cc:686] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc [term 1 LEADER]: Becoming Leader. State: Replica: 28382a9f1b774d13be0e43755ab647bc, State: Running, Role: LEADER
I20260926 13:48:43.492393 31025 consensus_queue.cc:237] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56099527e1d24e159dd7611641ade1a9" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39051 } } peers { permanent_uuid: "5140ef4e2fd34aaa969943c38aa693e5" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44849 } } peers { permanent_uuid: "28382a9f1b774d13be0e43755ab647bc" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35933 } }
I20260926 13:48:43.495239 30550 catalog_manager.cc:5495] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 28382a9f1b774d13be0e43755ab647bc (127.12.170.66). New cstate: current_term: 1 leader_uuid: "28382a9f1b774d13be0e43755ab647bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56099527e1d24e159dd7611641ade1a9" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5140ef4e2fd34aaa969943c38aa693e5" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28382a9f1b774d13be0e43755ab647bc" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 35933 } health_report { overall_health: HEALTHY } } }
I20260926 13:48:43.512092 31017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:43.509130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48980 (local address 127.12.170.66:35933)
0926 13:48:43.509247 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:43.509251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:43.509263 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:48:43.509379 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:43.509381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:43.509429 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:43.509506 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:48:43.509667 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.510134 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.510892 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:43.511088 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:43.511153 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:43.511350 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:48:43.511750 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:48:43.511843 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:43.511918 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 13:48:43.511958 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 13:48:43.517241 30693 raft_consensus.cc:1261] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 56099527e1d24e159dd7611641ade1a9 [term 1 FOLLOWER]: Refusing update from remote peer 28382a9f1b774d13be0e43755ab647bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:48:43.517246 30955 raft_consensus.cc:1261] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 5140ef4e2fd34aaa969943c38aa693e5 [term 1 FOLLOWER]: Refusing update from remote peer 28382a9f1b774d13be0e43755ab647bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:48:43.517858 31028 consensus_queue.cc:1035] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5140ef4e2fd34aaa969943c38aa693e5" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:48:43.518018 31025 consensus_queue.cc:1035] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "56099527e1d24e159dd7611641ade1a9" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 39051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:48:43.526526 31032 mvcc.cc:205] Tried to move back new op lower bound from 7333603424320122880 to 7333603424227065856. Current Snapshot: MvccSnapshot[applied={T|T < 7333603424320122880}]
I20260926 13:48:43.531342 30550 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:48:43.531473 30550 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:48:43.533581 30550 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 449fa7c144a84151b0b4b401ba33216e: Sending DeleteTablet for 3 replicas of tablet ec2ed0dcdd5a4c1ea5631fb3eda01e20
I20260926 13:48:43.534235 30935 tablet_service.cc:1514] Processing DeleteTablet for tablet ec2ed0dcdd5a4c1ea5631fb3eda01e20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:48:43 UTC) from {username='slave'} at 127.0.0.1:57830
I20260926 13:48:43.534358 30673 tablet_service.cc:1514] Processing DeleteTablet for tablet ec2ed0dcdd5a4c1ea5631fb3eda01e20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:48:43 UTC) from {username='slave'} at 127.0.0.1:40274
I20260926 13:48:43.534719 31042 tablet_replica.cc:330] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 56099527e1d24e159dd7611641ade1a9: stopping tablet replica
I20260926 13:48:43.534806 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 30611
I20260926 13:48:43.534916 30804 tablet_service.cc:1514] Processing DeleteTablet for tablet ec2ed0dcdd5a4c1ea5631fb3eda01e20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:48:43 UTC) from {username='slave'} at 127.0.0.1:48966
I20260926 13:48:43.535013 31042 raft_consensus.cc:2229] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 56099527e1d24e159dd7611641ade1a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:48:43.535388 31043 tablet_replica.cc:330] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc: stopping tablet replica
I20260926 13:48:43.535719 31043 raft_consensus.cc:2229] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:48:43.536038 31043 raft_consensus.cc:2258] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:48:43.537319 31043 ts_tablet_manager.cc:1905] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:48:43.539834 31041 tablet_replica.cc:330] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 5140ef4e2fd34aaa969943c38aa693e5: stopping tablet replica
I20260926 13:48:43.539880 31043 ts_tablet_manager.cc:1918] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:48:43.539961 31043 log.cc:1196] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-1/wal/wals/ec2ed0dcdd5a4c1ea5631fb3eda01e20
I20260926 13:48:43.540218 31041 raft_consensus.cc:2229] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 5140ef4e2fd34aaa969943c38aa693e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:48:43.540333 31043 ts_tablet_manager.cc:1939] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 28382a9f1b774d13be0e43755ab647bc: Deleting consensus metadata
I20260926 13:48:43.540585 31041 raft_consensus.cc:2258] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 5140ef4e2fd34aaa969943c38aa693e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:48:43.541338 30537 catalog_manager.cc:4841] TS 28382a9f1b774d13be0e43755ab647bc (127.12.170.66:35933): tablet ec2ed0dcdd5a4c1ea5631fb3eda01e20 (table test-table [id=3cc6851d10d24d0fbb287632f69e8acc]) successfully deleted
I20260926 13:48:43.541893 31041 ts_tablet_manager.cc:1905] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 5140ef4e2fd34aaa969943c38aa693e5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 13:48:43.543049 30537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.170.65:39051 (error 108)
I20260926 13:48:43.543232 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 30742
I20260926 13:48:43.545514 31041 ts_tablet_manager.cc:1918] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 5140ef4e2fd34aaa969943c38aa693e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:48:43.545605 31041 log.cc:1196] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 5140ef4e2fd34aaa969943c38aa693e5: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/ec2ed0dcdd5a4c1ea5631fb3eda01e20
I20260926 13:48:43.545864 31041 ts_tablet_manager.cc:1939] T ec2ed0dcdd5a4c1ea5631fb3eda01e20 P 5140ef4e2fd34aaa969943c38aa693e5: Deleting consensus metadata
I20260926 13:48:43.546922 30536 catalog_manager.cc:4841] TS 5140ef4e2fd34aaa969943c38aa693e5 (127.12.170.67:44849): tablet ec2ed0dcdd5a4c1ea5631fb3eda01e20 (table test-table [id=3cc6851d10d24d0fbb287632f69e8acc]) successfully deleted
I20260926 13:48:43.547456 31012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:43.547200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:39051 (local address 127.0.0.1:40282)
0926 13:48:43.547315 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:43.547384 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.65:39051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 13:48:43.547598 30537 catalog_manager.cc:4584] TS 56099527e1d24e159dd7611641ade1a9 (127.12.170.65:39051): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ec2ed0dcdd5a4c1ea5631fb3eda01e20: Network error: Client connection negotiation failed: client connection to 127.12.170.65:39051: connect: Connection refused (error 111)
I20260926 13:48:43.549193 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 30873
I20260926 13:48:43.553599 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 30520
2026-09-26T13:48:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1086 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:48:43 dist-test-slave-hc1k krb5kdc[31047](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:48:43 dist-test-slave-hc1k krb5kdc[31047](info): set up 2 sockets
Sep 26 13:48:43 dist-test-slave-hc1k krb5kdc[31047](info): commencing operation
krb5kdc: starting...
W20260926 13:48:44.503244 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:48:44 dist-test-slave-hc1k krb5kdc[31047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:48:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:48:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:48:44.615015 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:45625
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:45625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:48:44.718498 31060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:44.718731 31060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:44.718776 31060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:44.722651 31060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:48:44.722698 31060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:44.722716 31060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:44.722730 31060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:48:44.722743 31060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:48:44.727677 31060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:45625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:45625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:44.728516 31060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:44.729384 31060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:48:44.737169 31060 server_base.cc:1034] running on GCE node
W20260926 13:48:44.737143 31066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:44.737151 31065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:44.737349 31068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:44.737855 31060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:44.738811 31060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:44.739964 31060 hybrid_clock.cc:648] HybridClock initialized: now 1790430524739949 us; error 34 us; skew 500 ppm
Sep 26 13:48:44 dist-test-slave-hc1k krb5kdc[31047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430524, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:44.742590 31060 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:48:44.743378 31060 webserver.cc:460] Webserver started at http://127.12.170.126:42811/ using document root <none> and password file <none>
I20260926 13:48:44.743902 31060 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:44.743966 31060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:44.744146 31060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:44.745999 31060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "ddd4f30d8ba8418bb5768120265c56f8"
format_stamp: "Formatted at 2026-09-26 13:48:44 on dist-test-slave-hc1k"
server_key: "420b94a76ef26da2003fa0cacbb171ce"
server_key_iv: "4261bac3c48bab7174a45b35df28c7f4"
server_key_version: "encryptionkey@0"
I20260926 13:48:44.746495 31060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "ddd4f30d8ba8418bb5768120265c56f8"
format_stamp: "Formatted at 2026-09-26 13:48:44 on dist-test-slave-hc1k"
server_key: "420b94a76ef26da2003fa0cacbb171ce"
server_key_iv: "4261bac3c48bab7174a45b35df28c7f4"
server_key_version: "encryptionkey@0"
I20260926 13:48:44.749850 31060 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260926 13:48:44.752058 31075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:44.753136 31060 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:48:44.753238 31060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "ddd4f30d8ba8418bb5768120265c56f8"
format_stamp: "Formatted at 2026-09-26 13:48:44 on dist-test-slave-hc1k"
server_key: "420b94a76ef26da2003fa0cacbb171ce"
server_key_iv: "4261bac3c48bab7174a45b35df28c7f4"
server_key_version: "encryptionkey@0"
I20260926 13:48:44.753345 31060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:44.766781 31060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:44.769510 31060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:44.769659 31060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:44.775748 31060 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:45625
I20260926 13:48:44.775802 31127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:45625 every 8 connection(s)
I20260926 13:48:44.776584 31060 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:48:44.779147 31128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:44.780306 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 31060
I20260926 13:48:44.780440 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:48:44.780799 12969 external_mini_cluster.cc:1428] Setting key 6821be8d44d847882a158ae0e19b5be4
I20260926 13:48:44.785707 31128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddd4f30d8ba8418bb5768120265c56f8: Bootstrap starting.
I20260926 13:48:44.788132 31128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddd4f30d8ba8418bb5768120265c56f8: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:44.788839 31128 log.cc:824] T 00000000000000000000000000000000 P ddd4f30d8ba8418bb5768120265c56f8: Log is configured to *not* fsync() on all Append() calls
Sep 26 13:48:44 dist-test-slave-hc1k krb5kdc[31047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:48:44.790751 31128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddd4f30d8ba8418bb5768120265c56f8: No bootstrap required, opened a new log
I20260926 13:48:44.795083 31131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:44.782054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55460 (local address 127.12.170.126:45625)
0926 13:48:44.782442 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:44.782465 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:44.782507 (+ 42us) server_negotiation.cc:407] Connection header received
0926 13:48:44.783313 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:44.783336 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:44.783762 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:44.784101 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:44.784787 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:44.786183 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:44.787023 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:44.787726 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:44.787884 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:44.790213 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:44.790219 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:44.790233 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:44.790265 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:44.792186 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:44.792599 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:44.792605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:44.792611 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:44.792767 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:44.793003 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:44.793007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:44.793009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:44.793511 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:44.793698 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:44.793877 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 13:48:44.794158 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":100,"threads_started":1}
I20260926 13:48:44.795203 31128 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddd4f30d8ba8418bb5768120265c56f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddd4f30d8ba8418bb5768120265c56f8" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45625 } }
I20260926 13:48:44.795452 31128 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddd4f30d8ba8418bb5768120265c56f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:44.795511 31128 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddd4f30d8ba8418bb5768120265c56f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddd4f30d8ba8418bb5768120265c56f8, State: Initialized, Role: FOLLOWER
I20260926 13:48:44.795976 31128 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddd4f30d8ba8418bb5768120265c56f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddd4f30d8ba8418bb5768120265c56f8" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45625 } }
I20260926 13:48:44.796136 31128 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddd4f30d8ba8418bb5768120265c56f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:48:44.796219 31128 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddd4f30d8ba8418bb5768120265c56f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:48:44.796317 31128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddd4f30d8ba8418bb5768120265c56f8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:44.797199 31128 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddd4f30d8ba8418bb5768120265c56f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddd4f30d8ba8418bb5768120265c56f8" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45625 } }
I20260926 13:48:44.797521 31128 leader_election.cc:302] T 00000000000000000000000000000000 P ddd4f30d8ba8418bb5768120265c56f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddd4f30d8ba8418bb5768120265c56f8; no voters:
I20260926 13:48:44.797847 31128 leader_election.cc:288] T 00000000000000000000000000000000 P ddd4f30d8ba8418bb5768120265c56f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:48:44.797935 31133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddd4f30d8ba8418bb5768120265c56f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:48:44.798151 31133 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddd4f30d8ba8418bb5768120265c56f8 [term 1 LEADER]: Becoming Leader. State: Replica: ddd4f30d8ba8418bb5768120265c56f8, State: Running, Role: LEADER
I20260926 13:48:44.798475 31133 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddd4f30d8ba8418bb5768120265c56f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddd4f30d8ba8418bb5768120265c56f8" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45625 } }
I20260926 13:48:44.798902 31128 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddd4f30d8ba8418bb5768120265c56f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:48:44.800017 31134 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd4f30d8ba8418bb5768120265c56f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddd4f30d8ba8418bb5768120265c56f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddd4f30d8ba8418bb5768120265c56f8" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45625 } } }
I20260926 13:48:44.800028 31135 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddd4f30d8ba8418bb5768120265c56f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddd4f30d8ba8418bb5768120265c56f8. Latest consensus state: current_term: 1 leader_uuid: "ddd4f30d8ba8418bb5768120265c56f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddd4f30d8ba8418bb5768120265c56f8" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45625 } } }
I20260926 13:48:44.800163 31134 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd4f30d8ba8418bb5768120265c56f8 [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:44.800166 31135 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddd4f30d8ba8418bb5768120265c56f8 [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:44.800557 31142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:48:44.803547 31142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:48:44.808426 31142 catalog_manager.cc:1347] Generated new cluster ID: cd23ea82bd65471eba4ffcc75f45b6bd
I20260926 13:48:44.808487 31142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:48:44.860250 31142 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:48:44.861563 31142 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:48:44.875054 31142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddd4f30d8ba8418bb5768120265c56f8: Generated new TSK 0
I20260926 13:48:44.875880 31142 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.170.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
I20260926 13:48:44.902295 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:0
--local_ip_for_outbound_sockets=127.12.170.65
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:45625
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:48:45.005968 31154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:45.006214 31154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:45.006273 31154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:45.010172 31154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:45.010224 31154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:45.010310 31154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:48:45.015286 31154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:45625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:45.016217 31154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:45.017138 31154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:45.025946 31162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:45.025918 31160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:45.025919 31159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:45.026046 31154 server_base.cc:1034] running on GCE node
I20260926 13:48:45.026544 31154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:45.027132 31154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:45.028290 31154 hybrid_clock.cc:648] HybridClock initialized: now 1790430525028270 us; error 31 us; skew 500 ppm
Sep 26 13:48:45 dist-test-slave-hc1k krb5kdc[31047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430525, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:45.030766 31154 init.cc:377] Logged in from keytab as kudu/127.12.170.65@KRBTEST.COM (short username kudu)
I20260926 13:48:45.031512 31154 webserver.cc:460] Webserver started at http://127.12.170.65:43583/ using document root <none> and password file <none>
I20260926 13:48:45.032022 31154 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:45.032081 31154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:45.032281 31154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:45.034132 31154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-0/data/instance:
uuid: "59e31e3c82fb42c99102047f07709557"
format_stamp: "Formatted at 2026-09-26 13:48:45 on dist-test-slave-hc1k"
server_key: "e7b6904680d6c1af16885dc6626c5a0f"
server_key_iv: "8d93d5d56dabe645927e48a48f88a575"
server_key_version: "encryptionkey@0"
I20260926 13:48:45.034636 31154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance:
uuid: "59e31e3c82fb42c99102047f07709557"
format_stamp: "Formatted at 2026-09-26 13:48:45 on dist-test-slave-hc1k"
server_key: "e7b6904680d6c1af16885dc6626c5a0f"
server_key_iv: "8d93d5d56dabe645927e48a48f88a575"
server_key_version: "encryptionkey@0"
I20260926 13:48:45.037987 31154 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:48:45.040161 31169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:45.041285 31154 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:48:45.041373 31154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "59e31e3c82fb42c99102047f07709557"
format_stamp: "Formatted at 2026-09-26 13:48:45 on dist-test-slave-hc1k"
server_key: "e7b6904680d6c1af16885dc6626c5a0f"
server_key_iv: "8d93d5d56dabe645927e48a48f88a575"
server_key_version: "encryptionkey@0"
I20260926 13:48:45.041464 31154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:45.063966 31154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:45.066803 31154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:45.066958 31154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:45.067482 31154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:48:45.068302 31154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:48:45.068347 31154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:45.068385 31154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:48:45.068413 31154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:45.076339 31154 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:40029
I20260926 13:48:45.076375 31282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:40029 every 8 connection(s)
I20260926 13:48:45.077123 31154 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:48:45.077844 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 31154
I20260926 13:48:45.077953 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance
I20260926 13:48:45.078269 12969 external_mini_cluster.cc:1428] Setting key cd9cba6caafceb853ca277ec48467025
WARNING: no policy specified for kudu/127.12.170.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.66@KRBTEST.COM" created.
Sep 26 13:48:45 dist-test-slave-hc1k krb5kdc[31047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430525, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:48:45.095191 31131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:45.078894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:48747 (local address 127.12.170.126:45625)
0926 13:48:45.079011 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:45.079015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:45.081407 (+ 2392us) server_negotiation.cc:407] Connection header received
0926 13:48:45.082558 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:45.082561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:45.082612 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:45.082708 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:45.084135 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.084679 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.086323 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.086550 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.086628 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:45.091393 (+ 4765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:45.091396 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:45.091398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:45.091423 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:45.092930 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:45.093448 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:45.093452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:45.093453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:45.093550 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:45.093949 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:45.093953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:45.093983 (+ 30us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:45.094233 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:45.094335 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:45.094875 (+ 540us) server_negotiation.cc:299] Negotiation successful
0926 13:48:45.095014 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 13:48:45.095916 31285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:45.079268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:45625 (local address 127.12.170.65:48747)
0926 13:48:45.081246 (+ 1978us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:45.081297 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:45.082363 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:45.082903 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:45.082912 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:45.083328 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:48:45.083988 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:45.084010 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.084813 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.084817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:45.086198 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:45.086211 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.086697 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.086701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:45.086903 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:45.087473 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:48:45.087494 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:48:45.091199 (+ 3705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:48:45.093034 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:45.093039 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:45.093053 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:45.093308 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:45.093653 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:45.093657 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:45.093659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:45.093839 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:45.094371 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:48:45.094377 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:48:45.094636 (+ 259us) client_negotiation.cc:769] Sending connection context
0926 13:48:45.094785 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 13:48:45.095027 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":132,"threads_started":1}
I20260926 13:48:45.097654 31283 heartbeater.cc:347] Connected to a master server at 127.12.170.126:45625
I20260926 13:48:45.097878 31283 heartbeater.cc:464] Registering TS with master...
I20260926 13:48:45.098352 31283 heartbeater.cc:511] Master 127.12.170.126:45625 requested a full tablet report, sending...
I20260926 13:48:45.100108 31092 ts_manager.cc:194] Registered new tserver with Master: 59e31e3c82fb42c99102047f07709557 (127.12.170.65:40029)
I20260926 13:48:45.101339 31092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.65@KRBTEST.COM'} at 127.12.170.65:48747
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
I20260926 13:48:45.106338 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:0
--local_ip_for_outbound_sockets=127.12.170.66
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:45625
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:48:45.209348 31288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:45.209573 31288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:45.209613 31288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:45.213518 31288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:45.213570 31288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:45.213625 31288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:48:45.218614 31288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:45625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:45.219514 31288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:45.220449 31288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:45.229444 31296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:45.229508 31293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:45.229600 31294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:45.229501 31288 server_base.cc:1034] running on GCE node
I20260926 13:48:45.230024 31288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:45.230551 31288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:45.231711 31288 hybrid_clock.cc:648] HybridClock initialized: now 1790430525231692 us; error 38 us; skew 500 ppm
Sep 26 13:48:45 dist-test-slave-hc1k krb5kdc[31047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430525, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:45.234210 31288 init.cc:377] Logged in from keytab as kudu/127.12.170.66@KRBTEST.COM (short username kudu)
I20260926 13:48:45.235028 31288 webserver.cc:460] Webserver started at http://127.12.170.66:44199/ using document root <none> and password file <none>
I20260926 13:48:45.235546 31288 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:45.235608 31288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:45.235828 31288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:45.237701 31288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-1/data/instance:
uuid: "c30e03f72b344a079d1cf13743af6ae3"
format_stamp: "Formatted at 2026-09-26 13:48:45 on dist-test-slave-hc1k"
server_key: "cf8a63bd6890d7534bf7a5fc5439affc"
server_key_iv: "57c8a73aada9c947ee622ba5a8a7a4b2"
server_key_version: "encryptionkey@0"
I20260926 13:48:45.238247 31288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance:
uuid: "c30e03f72b344a079d1cf13743af6ae3"
format_stamp: "Formatted at 2026-09-26 13:48:45 on dist-test-slave-hc1k"
server_key: "cf8a63bd6890d7534bf7a5fc5439affc"
server_key_iv: "57c8a73aada9c947ee622ba5a8a7a4b2"
server_key_version: "encryptionkey@0"
I20260926 13:48:45.241652 31288 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:48:45.243963 31303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:45.245206 31288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 13:48:45.245301 31288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "c30e03f72b344a079d1cf13743af6ae3"
format_stamp: "Formatted at 2026-09-26 13:48:45 on dist-test-slave-hc1k"
server_key: "cf8a63bd6890d7534bf7a5fc5439affc"
server_key_iv: "57c8a73aada9c947ee622ba5a8a7a4b2"
server_key_version: "encryptionkey@0"
I20260926 13:48:45.245388 31288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:45.262502 31288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:45.265318 31288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:45.265481 31288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:45.266021 31288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:48:45.266786 31288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:48:45.266830 31288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:45.266881 31288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:48:45.266906 31288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:45.275405 31288 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:44825
I20260926 13:48:45.275478 31416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:44825 every 8 connection(s)
I20260926 13:48:45.276300 31288 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:48:45.281692 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 31288
I20260926 13:48:45.281821 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance
I20260926 13:48:45.282105 12969 external_mini_cluster.cc:1428] Setting key e5a0499742bafd7961dd8fd67e1385d6
Sep 26 13:48:45 dist-test-slave-hc1k krb5kdc[31047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430525, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:48:45.289738 31131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:45.277987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:37987 (local address 127.12.170.126:45625)
0926 13:48:45.278122 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:45.278126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:45.278920 (+ 794us) server_negotiation.cc:407] Connection header received
0926 13:48:45.280196 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:45.280200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:45.280258 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:45.280332 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:45.281475 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.282232 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.283452 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.283790 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.283896 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:45.286131 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:45.286134 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:45.286136 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:45.286160 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:45.287510 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:45.288094 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:45.288096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:45.288097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:45.288175 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:45.288572 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:45.288575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:45.288576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:45.288773 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:45.288858 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:45.289472 (+ 614us) server_negotiation.cc:299] Negotiation successful
0926 13:48:45.289608 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 13:48:45.290807 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:45.278339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:45625 (local address 127.12.170.66:37987)
0926 13:48:45.278765 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:45.278807 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:45.279985 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:45.280512 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:45.280519 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:45.280849 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:48:45.281317 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:45.281331 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.282354 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.282357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:45.283336 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:45.283347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.283920 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.283923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:45.284067 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:45.284590 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:48:45.284605 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:48:45.285983 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:48:45.287628 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:45.287635 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:45.287671 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:45.288002 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:45.288270 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:45.288273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:45.288275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:45.288451 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:45.288876 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:48:45.288883 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:48:45.289194 (+ 311us) client_negotiation.cc:769] Sending connection context
0926 13:48:45.289415 (+ 221us) client_negotiation.cc:240] Negotiation successful
0926 13:48:45.289761 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.12.170.67@KRBTEST.COM; defaulting to no policy
I20260926 13:48:45.291918 31417 heartbeater.cc:347] Connected to a master server at 127.12.170.126:45625
I20260926 13:48:45.292138 31417 heartbeater.cc:464] Registering TS with master...
I20260926 13:48:45.292600 31417 heartbeater.cc:511] Master 127.12.170.126:45625 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.67@KRBTEST.COM" created.
I20260926 13:48:45.293612 31092 ts_manager.cc:194] Registered new tserver with Master: c30e03f72b344a079d1cf13743af6ae3 (127.12.170.66:44825)
I20260926 13:48:45.294198 31092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.66@KRBTEST.COM'} at 127.12.170.66:37987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
I20260926 13:48:45.308080 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:0
--local_ip_for_outbound_sockets=127.12.170.67
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:45625
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:41053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:48:45.411502 31422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:45.411757 31422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:45.411796 31422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:45.415678 31422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:45.415728 31422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:45.415782 31422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:48:45.420706 31422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:41053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:45625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:45.421646 31422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:45.422542 31422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:45.431515 31430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:45.431591 31422 server_base.cc:1034] running on GCE node
W20260926 13:48:45.431499 31428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:45.431525 31427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:45.432194 31422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:45.432734 31422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:45.433887 31422 hybrid_clock.cc:648] HybridClock initialized: now 1790430525433869 us; error 28 us; skew 500 ppm
Sep 26 13:48:45 dist-test-slave-hc1k krb5kdc[31047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430525, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:45.439268 31422 init.cc:377] Logged in from keytab as kudu/127.12.170.67@KRBTEST.COM (short username kudu)
I20260926 13:48:45.440084 31422 webserver.cc:460] Webserver started at http://127.12.170.67:41277/ using document root <none> and password file <none>
I20260926 13:48:45.440588 31422 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:45.440644 31422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:45.440830 31422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:45.442744 31422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-2/data/instance:
uuid: "e49a265a6ea3456daa6001be6f5f6171"
format_stamp: "Formatted at 2026-09-26 13:48:45 on dist-test-slave-hc1k"
server_key: "febbda1952b30aab51d4a7cde616d06b"
server_key_iv: "4681d370349fc39f1f5ac042c33e2f4b"
server_key_version: "encryptionkey@0"
I20260926 13:48:45.443286 31422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance:
uuid: "e49a265a6ea3456daa6001be6f5f6171"
format_stamp: "Formatted at 2026-09-26 13:48:45 on dist-test-slave-hc1k"
server_key: "febbda1952b30aab51d4a7cde616d06b"
server_key_iv: "4681d370349fc39f1f5ac042c33e2f4b"
server_key_version: "encryptionkey@0"
I20260926 13:48:45.446843 31422 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 13:48:45.449204 31437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:45.450341 31422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:48:45.450439 31422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "e49a265a6ea3456daa6001be6f5f6171"
format_stamp: "Formatted at 2026-09-26 13:48:45 on dist-test-slave-hc1k"
server_key: "febbda1952b30aab51d4a7cde616d06b"
server_key_iv: "4681d370349fc39f1f5ac042c33e2f4b"
server_key_version: "encryptionkey@0"
I20260926 13:48:45.450523 31422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:45.471786 31422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:45.474665 31422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:45.474824 31422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:45.475342 31422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:48:45.476135 31422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:48:45.476181 31422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:45.476238 31422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:48:45.476264 31422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:45.484252 31422 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:42805
I20260926 13:48:45.484272 31550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:42805 every 8 connection(s)
I20260926 13:48:45.485049 31422 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
Sep 26 13:48:45 dist-test-slave-hc1k krb5kdc[31047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430525, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:48:45.493826 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 31422
I20260926 13:48:45.493933 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance
I20260926 13:48:45.494202 12969 external_mini_cluster.cc:1428] Setting key d491f033789920817bfe8de7cc3cfa41
I20260926 13:48:45.497522 31131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:45.486707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:45415 (local address 127.12.170.126:45625)
0926 13:48:45.486854 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:45.486859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:45.487670 (+ 811us) server_negotiation.cc:407] Connection header received
0926 13:48:45.488867 (+ 1197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:45.488870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:45.488914 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:45.488985 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:45.490003 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.490519 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.491616 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.491879 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.491956 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:45.494166 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:45.494169 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:45.494171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:45.494194 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:45.495666 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:45.496139 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:45.496143 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:45.496144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:45.496236 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:45.496546 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:45.496549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:45.496551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:45.496726 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:45.496805 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:45.497259 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 13:48:45.497370 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:48:45.498255 31553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:45.487087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:45625 (local address 127.12.170.67:45415)
0926 13:48:45.487513 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:45.487552 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:45.488671 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:45.489123 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:45.489129 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:45.489438 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:48:45.489894 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:45.489907 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.490623 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.490626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:45.491501 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:45.491510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.492005 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.492008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:45.492138 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:45.492597 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:48:45.492610 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:48:45.494017 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:48:45.495757 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:45.495761 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:45.495772 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:45.496005 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:45.496315 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:45.496326 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:45.496329 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:45.496459 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:45.496816 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:48:45.496822 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:48:45.497047 (+ 225us) client_negotiation.cc:769] Sending connection context
0926 13:48:45.497203 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 13:48:45.497511 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":107,"threads_started":1}
I20260926 13:48:45.499253 31551 heartbeater.cc:347] Connected to a master server at 127.12.170.126:45625
I20260926 13:48:45.499504 31551 heartbeater.cc:464] Registering TS with master...
I20260926 13:48:45.500015 31551 heartbeater.cc:511] Master 127.12.170.126:45625 requested a full tablet report, sending...
I20260926 13:48:45.501027 31092 ts_manager.cc:194] Registered new tserver with Master: e49a265a6ea3456daa6001be6f5f6171 (127.12.170.67:42805)
I20260926 13:48:45.501611 31092 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.67@KRBTEST.COM'} at 127.12.170.67:45415
I20260926 13:48:45.507705 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:48:45.516014 31131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:45.509500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55476 (local address 127.12.170.126:45625)
0926 13:48:45.509623 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:45.509627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:45.509743 (+ 116us) server_negotiation.cc:407] Connection header received
0926 13:48:45.509847 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:45.509849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:45.509884 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:45.509943 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:45.510591 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.511073 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.511916 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.512111 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.512179 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:45.513546 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:45.513549 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:45.513550 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:45.513570 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:45.514604 (+ 1034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:45.514994 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:45.514996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:45.514997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:45.515057 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:45.515300 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:45.515303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:45.515304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:45.515469 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:45.515572 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:45.515789 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 13:48:45.515870 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 13:48:45.518412 31092 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:48:45.521559 31092 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:48:45.535004 31567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:45.530262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44346 (local address 127.12.170.65:40029)
0926 13:48:45.530940 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:45.530951 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:45.530973 (+ 22us) server_negotiation.cc:407] Connection header received
0926 13:48:45.531035 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:45.531038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:45.531166 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:45.531292 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:45.531487 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.532418 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.533802 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.534319 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.534615 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:45.534744 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:45.534792 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 13:48:45.534874 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":96,"threads_started":1}
I20260926 13:48:45.535008 31565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:45.529793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:40029 (local address 127.0.0.1:44346)
0926 13:48:45.530600 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:45.530613 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:45.530683 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:45.531281 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:45.531284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:45.531300 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:45.531391 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:45.531398 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.532547 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.532550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:45.533709 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:45.533716 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.534426 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.534429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:45.534592 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:45.534636 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 13:48:45.534727 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 13:48:45.534837 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":75,"thread_start_us":77,"threads_started":1}
I20260926 13:48:45.535066 31562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:45.529216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:44825 (local address 127.0.0.1:56676)
0926 13:48:45.529810 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:45.529822 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:45.529885 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:45.530605 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:45.530608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:45.530628 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:45.530770 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:45.530777 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.532135 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.532138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:45.533354 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:45.533364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.534457 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.534461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:45.534808 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:45.534829 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 13:48:45.534883 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 13:48:45.534929 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":295,"thread_start_us":84,"threads_started":1}
I20260926 13:48:45.535377 31564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:45.529250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41896 (local address 127.12.170.67:42805)
0926 13:48:45.529671 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:45.529682 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:45.529947 (+ 265us) server_negotiation.cc:407] Connection header received
0926 13:48:45.530028 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:45.530034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:45.530253 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:45.530405 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:45.531009 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.532106 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.533602 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.534374 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.534462 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:45.534603 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:45.535137 (+ 534us) server_negotiation.cc:299] Negotiation successful
0926 13:48:45.535249 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":104,"threads_started":1}
I20260926 13:48:45.535441 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:45.529854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56676 (local address 127.12.170.66:44825)
0926 13:48:45.530130 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:45.530140 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:45.530162 (+ 22us) server_negotiation.cc:407] Connection header received
0926 13:48:45.530235 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:45.530240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:45.530453 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:45.530707 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:45.530841 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.532008 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.533475 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.534315 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.534577 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:45.534714 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:45.535229 (+ 515us) server_negotiation.cc:299] Negotiation successful
0926 13:48:45.535322 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":115,"threads_started":1}
I20260926 13:48:45.535075 31563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:45.529105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:42805 (local address 127.0.0.1:41896)
0926 13:48:45.529623 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:45.529670 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:45.529809 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:45.530564 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:45.530570 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:45.530591 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:45.530898 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:45.530903 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.532213 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.532216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:45.533489 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:45.533499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.534658 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.534662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:45.534740 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:45.534764 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 13:48:45.534830 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 13:48:45.534896 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":32,"spinlock_wait_cycles":3968,"thread_start_us":78,"threads_started":1}
I20260926 13:48:45.537962 31485 tablet_service.cc:1467] Processing CreateTablet for tablet 2442ddceaabf4e6ebf60526709a69853 (DEFAULT_TABLE table=test-table [id=ce4e77aa3e244285b72ebf931a17c05c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:48:45.538081 31351 tablet_service.cc:1467] Processing CreateTablet for tablet 2442ddceaabf4e6ebf60526709a69853 (DEFAULT_TABLE table=test-table [id=ce4e77aa3e244285b72ebf931a17c05c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:48:45.538082 31217 tablet_service.cc:1467] Processing CreateTablet for tablet 2442ddceaabf4e6ebf60526709a69853 (DEFAULT_TABLE table=test-table [id=ce4e77aa3e244285b72ebf931a17c05c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:48:45.538815 31485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2442ddceaabf4e6ebf60526709a69853. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:45.539014 31351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2442ddceaabf4e6ebf60526709a69853. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:45.539014 31217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2442ddceaabf4e6ebf60526709a69853. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:45.544835 31568 tablet_bootstrap.cc:492] T 2442ddceaabf4e6ebf60526709a69853 P e49a265a6ea3456daa6001be6f5f6171: Bootstrap starting.
I20260926 13:48:45.546368 31570 tablet_bootstrap.cc:492] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557: Bootstrap starting.
I20260926 13:48:45.546730 31569 tablet_bootstrap.cc:492] T 2442ddceaabf4e6ebf60526709a69853 P c30e03f72b344a079d1cf13743af6ae3: Bootstrap starting.
I20260926 13:48:45.547008 31568 tablet_bootstrap.cc:654] T 2442ddceaabf4e6ebf60526709a69853 P e49a265a6ea3456daa6001be6f5f6171: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:45.548347 31568 log.cc:824] T 2442ddceaabf4e6ebf60526709a69853 P e49a265a6ea3456daa6001be6f5f6171: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:45.548446 31570 tablet_bootstrap.cc:654] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:45.548946 31569 tablet_bootstrap.cc:654] T 2442ddceaabf4e6ebf60526709a69853 P c30e03f72b344a079d1cf13743af6ae3: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:45.549615 31570 log.cc:824] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:45.550092 31568 tablet_bootstrap.cc:492] T 2442ddceaabf4e6ebf60526709a69853 P e49a265a6ea3456daa6001be6f5f6171: No bootstrap required, opened a new log
I20260926 13:48:45.550132 31569 log.cc:824] T 2442ddceaabf4e6ebf60526709a69853 P c30e03f72b344a079d1cf13743af6ae3: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:45.550287 31568 ts_tablet_manager.cc:1397] T 2442ddceaabf4e6ebf60526709a69853 P e49a265a6ea3456daa6001be6f5f6171: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 13:48:45.551417 31570 tablet_bootstrap.cc:492] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557: No bootstrap required, opened a new log
I20260926 13:48:45.551592 31570 ts_tablet_manager.cc:1397] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 13:48:45.551928 31569 tablet_bootstrap.cc:492] T 2442ddceaabf4e6ebf60526709a69853 P c30e03f72b344a079d1cf13743af6ae3: No bootstrap required, opened a new log
I20260926 13:48:45.552088 31569 ts_tablet_manager.cc:1397] T 2442ddceaabf4e6ebf60526709a69853 P c30e03f72b344a079d1cf13743af6ae3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260926 13:48:45.553325 31568 raft_consensus.cc:348] T 2442ddceaabf4e6ebf60526709a69853 P e49a265a6ea3456daa6001be6f5f6171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49a265a6ea3456daa6001be6f5f6171" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42805 } } peers { permanent_uuid: "c30e03f72b344a079d1cf13743af6ae3" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 44825 } } peers { permanent_uuid: "59e31e3c82fb42c99102047f07709557" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40029 } }
I20260926 13:48:45.553579 31568 raft_consensus.cc:374] T 2442ddceaabf4e6ebf60526709a69853 P e49a265a6ea3456daa6001be6f5f6171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:45.553630 31568 raft_consensus.cc:729] T 2442ddceaabf4e6ebf60526709a69853 P e49a265a6ea3456daa6001be6f5f6171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49a265a6ea3456daa6001be6f5f6171, State: Initialized, Role: FOLLOWER
I20260926 13:48:45.554122 31568 consensus_queue.cc:260] T 2442ddceaabf4e6ebf60526709a69853 P e49a265a6ea3456daa6001be6f5f6171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49a265a6ea3456daa6001be6f5f6171" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42805 } } peers { permanent_uuid: "c30e03f72b344a079d1cf13743af6ae3" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 44825 } } peers { permanent_uuid: "59e31e3c82fb42c99102047f07709557" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40029 } }
I20260926 13:48:45.555028 31568 ts_tablet_manager.cc:1428] T 2442ddceaabf4e6ebf60526709a69853 P e49a265a6ea3456daa6001be6f5f6171: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 13:48:45.555356 31551 heartbeater.cc:503] Master 127.12.170.126:45625 was elected leader, sending a full tablet report...
I20260926 13:48:45.555361 31570 raft_consensus.cc:348] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49a265a6ea3456daa6001be6f5f6171" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42805 } } peers { permanent_uuid: "c30e03f72b344a079d1cf13743af6ae3" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 44825 } } peers { permanent_uuid: "59e31e3c82fb42c99102047f07709557" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40029 } }
I20260926 13:48:45.555604 31570 raft_consensus.cc:374] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:45.555663 31570 raft_consensus.cc:729] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59e31e3c82fb42c99102047f07709557, State: Initialized, Role: FOLLOWER
I20260926 13:48:45.555545 31569 raft_consensus.cc:348] T 2442ddceaabf4e6ebf60526709a69853 P c30e03f72b344a079d1cf13743af6ae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49a265a6ea3456daa6001be6f5f6171" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42805 } } peers { permanent_uuid: "c30e03f72b344a079d1cf13743af6ae3" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 44825 } } peers { permanent_uuid: "59e31e3c82fb42c99102047f07709557" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40029 } }
I20260926 13:48:45.555815 31569 raft_consensus.cc:374] T 2442ddceaabf4e6ebf60526709a69853 P c30e03f72b344a079d1cf13743af6ae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:45.555866 31569 raft_consensus.cc:729] T 2442ddceaabf4e6ebf60526709a69853 P c30e03f72b344a079d1cf13743af6ae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c30e03f72b344a079d1cf13743af6ae3, State: Initialized, Role: FOLLOWER
I20260926 13:48:45.556098 31570 consensus_queue.cc:260] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49a265a6ea3456daa6001be6f5f6171" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42805 } } peers { permanent_uuid: "c30e03f72b344a079d1cf13743af6ae3" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 44825 } } peers { permanent_uuid: "59e31e3c82fb42c99102047f07709557" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40029 } }
I20260926 13:48:45.556212 31569 consensus_queue.cc:260] T 2442ddceaabf4e6ebf60526709a69853 P c30e03f72b344a079d1cf13743af6ae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49a265a6ea3456daa6001be6f5f6171" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42805 } } peers { permanent_uuid: "c30e03f72b344a079d1cf13743af6ae3" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 44825 } } peers { permanent_uuid: "59e31e3c82fb42c99102047f07709557" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40029 } }
I20260926 13:48:45.556739 31283 heartbeater.cc:503] Master 127.12.170.126:45625 was elected leader, sending a full tablet report...
I20260926 13:48:45.556999 31570 ts_tablet_manager.cc:1428] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260926 13:48:45.557013 31569 ts_tablet_manager.cc:1428] T 2442ddceaabf4e6ebf60526709a69853 P c30e03f72b344a079d1cf13743af6ae3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 13:48:45.557278 31417 heartbeater.cc:503] Master 127.12.170.126:45625 was elected leader, sending a full tablet report...
W20260926 13:48:45.578182 31284 tablet.cc:2369] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:48:45.637679 31575 raft_consensus.cc:482] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:48:45.637880 31575 raft_consensus.cc:504] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49a265a6ea3456daa6001be6f5f6171" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42805 } } peers { permanent_uuid: "c30e03f72b344a079d1cf13743af6ae3" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 44825 } } peers { permanent_uuid: "59e31e3c82fb42c99102047f07709557" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40029 } }
I20260926 13:48:45.639005 31575 leader_election.cc:288] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e49a265a6ea3456daa6001be6f5f6171 (127.12.170.67:42805), c30e03f72b344a079d1cf13743af6ae3 (127.12.170.66:44825)
I20260926 13:48:45.643767 31564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:45.639310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:42673 (local address 127.12.170.67:42805)
0926 13:48:45.639559 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:45.639562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:45.639574 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:48:45.639747 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:45.639750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:45.639803 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:45.639882 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:45.640155 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.640934 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.642551 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.643170 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.643286 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:45.643338 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:45.643556 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 13:48:45.643620 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260926 13:48:45.643767 31566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:45.639342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:57333 (local address 127.12.170.66:44825)
0926 13:48:45.639467 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:45.639472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:45.639670 (+ 198us) server_negotiation.cc:407] Connection header received
0926 13:48:45.639796 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:45.639799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:45.639846 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:45.639913 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:45.640154 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.640935 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.642551 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.643170 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.643286 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:45.643338 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:45.643556 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 13:48:45.643620 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 13:48:45.643828 31578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:45.639256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:44825 (local address 127.12.170.65:57333)
0926 13:48:45.639591 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:45.639601 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:45.639711 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:45.639923 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:45.639925 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:45.639958 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:45.640053 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:45.640059 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.641035 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.641039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:45.642421 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:45.642427 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.643299 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.643302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:45.643440 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:45.643463 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 13:48:45.643536 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 13:48:45.643623 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":134,"spinlock_wait_cycles":896,"thread_start_us":111,"threads_started":1}
I20260926 13:48:45.644176 31577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:45.639217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:42805 (local address 127.12.170.65:42673)
0926 13:48:45.639507 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:45.639522 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:45.639622 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:45.639891 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:45.639893 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:45.639911 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:45.640064 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:45.640069 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.641035 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.641039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:45.642414 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:45.642426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.643284 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.643288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:45.643420 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:45.643463 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 13:48:45.643536 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 13:48:45.643623 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":23,"spinlock_wait_cycles":4864,"thread_start_us":100,"threads_started":1}
I20260926 13:48:45.644475 31371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2442ddceaabf4e6ebf60526709a69853" candidate_uuid: "59e31e3c82fb42c99102047f07709557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30e03f72b344a079d1cf13743af6ae3" is_pre_election: true
I20260926 13:48:45.644735 31505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2442ddceaabf4e6ebf60526709a69853" candidate_uuid: "59e31e3c82fb42c99102047f07709557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e49a265a6ea3456daa6001be6f5f6171" is_pre_election: true
I20260926 13:48:45.644773 31371 raft_consensus.cc:2454] T 2442ddceaabf4e6ebf60526709a69853 P c30e03f72b344a079d1cf13743af6ae3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59e31e3c82fb42c99102047f07709557 in term 0.
I20260926 13:48:45.644994 31505 raft_consensus.cc:2454] T 2442ddceaabf4e6ebf60526709a69853 P e49a265a6ea3456daa6001be6f5f6171 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59e31e3c82fb42c99102047f07709557 in term 0.
I20260926 13:48:45.645251 31172 leader_election.cc:302] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59e31e3c82fb42c99102047f07709557, c30e03f72b344a079d1cf13743af6ae3; no voters:
I20260926 13:48:45.645514 31575 raft_consensus.cc:2781] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:48:45.645591 31575 raft_consensus.cc:482] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:48:45.645638 31575 raft_consensus.cc:3037] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:45.646615 31575 raft_consensus.cc:504] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49a265a6ea3456daa6001be6f5f6171" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42805 } } peers { permanent_uuid: "c30e03f72b344a079d1cf13743af6ae3" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 44825 } } peers { permanent_uuid: "59e31e3c82fb42c99102047f07709557" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40029 } }
I20260926 13:48:45.646934 31575 leader_election.cc:288] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557 [CANDIDATE]: Term 1 election: Requested vote from peers e49a265a6ea3456daa6001be6f5f6171 (127.12.170.67:42805), c30e03f72b344a079d1cf13743af6ae3 (127.12.170.66:44825)
I20260926 13:48:45.647270 31505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2442ddceaabf4e6ebf60526709a69853" candidate_uuid: "59e31e3c82fb42c99102047f07709557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e49a265a6ea3456daa6001be6f5f6171"
I20260926 13:48:45.647383 31505 raft_consensus.cc:3037] T 2442ddceaabf4e6ebf60526709a69853 P e49a265a6ea3456daa6001be6f5f6171 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:45.648562 31505 raft_consensus.cc:2454] T 2442ddceaabf4e6ebf60526709a69853 P e49a265a6ea3456daa6001be6f5f6171 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59e31e3c82fb42c99102047f07709557 in term 1.
I20260926 13:48:45.648792 31371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2442ddceaabf4e6ebf60526709a69853" candidate_uuid: "59e31e3c82fb42c99102047f07709557" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c30e03f72b344a079d1cf13743af6ae3"
I20260926 13:48:45.648842 31171 leader_election.cc:302] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59e31e3c82fb42c99102047f07709557, e49a265a6ea3456daa6001be6f5f6171; no voters:
I20260926 13:48:45.648890 31371 raft_consensus.cc:3037] T 2442ddceaabf4e6ebf60526709a69853 P c30e03f72b344a079d1cf13743af6ae3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:45.648994 31575 raft_consensus.cc:2781] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:48:45.649248 31575 raft_consensus.cc:686] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557 [term 1 LEADER]: Becoming Leader. State: Replica: 59e31e3c82fb42c99102047f07709557, State: Running, Role: LEADER
I20260926 13:48:45.649602 31575 consensus_queue.cc:237] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49a265a6ea3456daa6001be6f5f6171" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42805 } } peers { permanent_uuid: "c30e03f72b344a079d1cf13743af6ae3" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 44825 } } peers { permanent_uuid: "59e31e3c82fb42c99102047f07709557" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40029 } }
I20260926 13:48:45.650120 31371 raft_consensus.cc:2454] T 2442ddceaabf4e6ebf60526709a69853 P c30e03f72b344a079d1cf13743af6ae3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59e31e3c82fb42c99102047f07709557 in term 1.
I20260926 13:48:45.652769 31091 catalog_manager.cc:5495] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59e31e3c82fb42c99102047f07709557 (127.12.170.65). New cstate: current_term: 1 leader_uuid: "59e31e3c82fb42c99102047f07709557" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49a265a6ea3456daa6001be6f5f6171" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c30e03f72b344a079d1cf13743af6ae3" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59e31e3c82fb42c99102047f07709557" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40029 } health_report { overall_health: HEALTHY } } }
I20260926 13:48:45.688372 31567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:45.685260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44358 (local address 127.12.170.65:40029)
0926 13:48:45.685369 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:45.685373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:45.685385 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:48:45.685488 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:45.685490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:45.685542 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:45.685637 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:48:45.685820 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.686323 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.687109 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.687320 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.687397 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:45.687672 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:48:45.688064 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:48:45.688169 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:45.688230 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 13:48:45.688265 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 13:48:45.693797 31371 raft_consensus.cc:1261] T 2442ddceaabf4e6ebf60526709a69853 P c30e03f72b344a079d1cf13743af6ae3 [term 1 FOLLOWER]: Refusing update from remote peer 59e31e3c82fb42c99102047f07709557: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:48:45.693797 31505 raft_consensus.cc:1261] T 2442ddceaabf4e6ebf60526709a69853 P e49a265a6ea3456daa6001be6f5f6171 [term 1 FOLLOWER]: Refusing update from remote peer 59e31e3c82fb42c99102047f07709557: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:48:45.694468 31579 consensus_queue.cc:1035] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c30e03f72b344a079d1cf13743af6ae3" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:48:45.694648 31575 consensus_queue.cc:1035] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e49a265a6ea3456daa6001be6f5f6171" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:48:45.702021 31584 mvcc.cc:205] Tried to move back new op lower bound from 7333603433235427328 to 7333603433063817216. Current Snapshot: MvccSnapshot[applied={T|T < 7333603433235427328}]
I20260926 13:48:45.702627 31583 mvcc.cc:205] Tried to move back new op lower bound from 7333603433235427328 to 7333603433063817216. Current Snapshot: MvccSnapshot[applied={T|T < 7333603433235427328}]
I20260926 13:48:45.711303 31092 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:48:45.711450 31092 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:48:45.713624 31092 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ddd4f30d8ba8418bb5768120265c56f8: Sending DeleteTablet for 3 replicas of tablet 2442ddceaabf4e6ebf60526709a69853
I20260926 13:48:45.714313 31485 tablet_service.cc:1514] Processing DeleteTablet for tablet 2442ddceaabf4e6ebf60526709a69853 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:48:45 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:41896
I20260926 13:48:45.714310 31351 tablet_service.cc:1514] Processing DeleteTablet for tablet 2442ddceaabf4e6ebf60526709a69853 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:48:45 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:56676
I20260926 13:48:45.714425 31217 tablet_service.cc:1514] Processing DeleteTablet for tablet 2442ddceaabf4e6ebf60526709a69853 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:48:45 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:44346
I20260926 13:48:45.714877 31594 tablet_replica.cc:330] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557: stopping tablet replica
I20260926 13:48:45.714885 31592 tablet_replica.cc:330] T 2442ddceaabf4e6ebf60526709a69853 P e49a265a6ea3456daa6001be6f5f6171: stopping tablet replica
I20260926 13:48:45.715181 31592 raft_consensus.cc:2229] T 2442ddceaabf4e6ebf60526709a69853 P e49a265a6ea3456daa6001be6f5f6171 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:48:45.715480 31594 raft_consensus.cc:2229] T 2442ddceaabf4e6ebf60526709a69853 P 59e31e3c82fb42c99102047f07709557 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:48:45.715561 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 31154
I20260926 13:48:45.717379 31593 tablet_replica.cc:330] T 2442ddceaabf4e6ebf60526709a69853 P c30e03f72b344a079d1cf13743af6ae3: stopping tablet replica
I20260926 13:48:45.717665 31593 raft_consensus.cc:2229] T 2442ddceaabf4e6ebf60526709a69853 P c30e03f72b344a079d1cf13743af6ae3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:48:45.717918 31593 raft_consensus.cc:2258] T 2442ddceaabf4e6ebf60526709a69853 P c30e03f72b344a079d1cf13743af6ae3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:48:45.715485 31592 raft_consensus.cc:2258] T 2442ddceaabf4e6ebf60526709a69853 P e49a265a6ea3456daa6001be6f5f6171 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:48:45.719120 31593 ts_tablet_manager.cc:1905] T 2442ddceaabf4e6ebf60526709a69853 P c30e03f72b344a079d1cf13743af6ae3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:48:45.721230 31593 ts_tablet_manager.cc:1918] T 2442ddceaabf4e6ebf60526709a69853 P c30e03f72b344a079d1cf13743af6ae3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:48:45.721318 31593 log.cc:1196] T 2442ddceaabf4e6ebf60526709a69853 P c30e03f72b344a079d1cf13743af6ae3: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-1/wal/wals/2442ddceaabf4e6ebf60526709a69853
I20260926 13:48:45.721616 31593 ts_tablet_manager.cc:1939] T 2442ddceaabf4e6ebf60526709a69853 P c30e03f72b344a079d1cf13743af6ae3: Deleting consensus metadata
I20260926 13:48:45.721778 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 31288
W20260926 13:48:45.721906 31077 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.170.65:40029 (error 108)
I20260926 13:48:45.722906 31592 ts_tablet_manager.cc:1905] T 2442ddceaabf4e6ebf60526709a69853 P e49a265a6ea3456daa6001be6f5f6171: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:48:45.724731 31563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:45.724433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:40029 (local address 127.0.0.1:44374)
0926 13:48:45.724570 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:45.724645 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.65:40029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 13:48:45.724990 31077 catalog_manager.cc:4584] TS 59e31e3c82fb42c99102047f07709557 (127.12.170.65:40029): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2442ddceaabf4e6ebf60526709a69853: Network error: Client connection negotiation failed: client connection to 127.12.170.65:40029: connect: Connection refused (error 111)
I20260926 13:48:45.725384 31592 ts_tablet_manager.cc:1918] T 2442ddceaabf4e6ebf60526709a69853 P e49a265a6ea3456daa6001be6f5f6171: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:48:45.725466 31592 log.cc:1196] T 2442ddceaabf4e6ebf60526709a69853 P e49a265a6ea3456daa6001be6f5f6171: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/2442ddceaabf4e6ebf60526709a69853
I20260926 13:48:45.725714 31592 ts_tablet_manager.cc:1939] T 2442ddceaabf4e6ebf60526709a69853 P e49a265a6ea3456daa6001be6f5f6171: Deleting consensus metadata
I20260926 13:48:45.726711 31077 catalog_manager.cc:4841] TS e49a265a6ea3456daa6001be6f5f6171 (127.12.170.67:42805): tablet 2442ddceaabf4e6ebf60526709a69853 (table test-table [id=ce4e77aa3e244285b72ebf931a17c05c]) successfully deleted
I20260926 13:48:45.727057 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 31422
I20260926 13:48:45.727834 31563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:45.727175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:44825 (local address 127.0.0.1:56686)
0926 13:48:45.727702 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:45.727754 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.66:44825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440}
I20260926 13:48:45.731755 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 31060
2026-09-26T13:48:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2179 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T13:48:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:48:45Z Disabled control of system clock
I20260926 13:48:45.759513 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:42493
--webserver_interface=127.12.170.126
--webserver_port=0
--builtin_ntp_servers=127.12.170.84:43073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:42493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:48:45.863081 31599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:45.863313 31599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:45.863345 31599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:45.867266 31599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:48:45.867319 31599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:45.867333 31599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:45.867349 31599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:48:45.867377 31599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:48:45.871920 31599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:43073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:42493
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:42493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:45.872972 31599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:45.874250 31599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:45.882186 31605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:45.882292 31607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:45.882301 31604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:45.882403 31599 server_base.cc:1034] running on GCE node
I20260926 13:48:45.883010 31599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:45.884057 31599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:45.885234 31599 hybrid_clock.cc:648] HybridClock initialized: now 1790430525885212 us; error 32 us; skew 500 ppm
I20260926 13:48:45.887012 31599 webserver.cc:460] Webserver started at http://127.12.170.126:45719/ using document root <none> and password file <none>
I20260926 13:48:45.887543 31599 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:45.887600 31599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:45.887802 31599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:45.889710 31599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "a8ed181fde43429a904ec632a4a91f8f"
format_stamp: "Formatted at 2026-09-26 13:48:45 on dist-test-slave-hc1k"
server_key: "4d3631bdf3bb7fb3b22b92b946c14c9c"
server_key_iv: "986abd3d8f56855a2b17f2023bc4faab"
server_key_version: "encryptionkey@0"
I20260926 13:48:45.890218 31599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "a8ed181fde43429a904ec632a4a91f8f"
format_stamp: "Formatted at 2026-09-26 13:48:45 on dist-test-slave-hc1k"
server_key: "4d3631bdf3bb7fb3b22b92b946c14c9c"
server_key_iv: "986abd3d8f56855a2b17f2023bc4faab"
server_key_version: "encryptionkey@0"
I20260926 13:48:45.893642 31599 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:48:45.895936 31613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:45.897150 31599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 13:48:45.897253 31599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "a8ed181fde43429a904ec632a4a91f8f"
format_stamp: "Formatted at 2026-09-26 13:48:45 on dist-test-slave-hc1k"
server_key: "4d3631bdf3bb7fb3b22b92b946c14c9c"
server_key_iv: "986abd3d8f56855a2b17f2023bc4faab"
server_key_version: "encryptionkey@0"
I20260926 13:48:45.897369 31599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:45.915623 31599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:45.916355 31599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:45.916524 31599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:45.923067 31599 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:42493
I20260926 13:48:45.923091 31665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:42493 every 8 connection(s)
I20260926 13:48:45.923975 31599 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:48:45.924279 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 31599
I20260926 13:48:45.924412 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:48:45.924803 12969 external_mini_cluster.cc:1428] Setting key 671c1b97d99155999801b8936ceb66b6
I20260926 13:48:45.927521 31666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:45.933102 31666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8ed181fde43429a904ec632a4a91f8f: Bootstrap starting.
I20260926 13:48:45.933727 31669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:45.926250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47336 (local address 127.12.170.126:42493)
0926 13:48:45.926909 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:45.926923 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:45.926957 (+ 34us) server_negotiation.cc:407] Connection header received
0926 13:48:45.927591 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:45.927613 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:45.928043 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:45.928374 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:45.928811 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.929732 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.930596 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:45.931051 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:45.931187 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:45.932123 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:45.932128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:45.932140 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:48:45.932505 (+ 365us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:45.932558 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:48:45.932577 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:45.932662 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:45.932796 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 13:48:45.933036 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":148,"threads_started":1}
I20260926 13:48:45.935286 31666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8ed181fde43429a904ec632a4a91f8f: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:45.936098 31666 log.cc:824] T 00000000000000000000000000000000 P a8ed181fde43429a904ec632a4a91f8f: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:45.937942 31666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8ed181fde43429a904ec632a4a91f8f: No bootstrap required, opened a new log
I20260926 13:48:45.941412 31666 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8ed181fde43429a904ec632a4a91f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ed181fde43429a904ec632a4a91f8f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 42493 } }
I20260926 13:48:45.941624 31666 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8ed181fde43429a904ec632a4a91f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:45.941684 31666 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8ed181fde43429a904ec632a4a91f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8ed181fde43429a904ec632a4a91f8f, State: Initialized, Role: FOLLOWER
I20260926 13:48:45.942085 31666 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8ed181fde43429a904ec632a4a91f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ed181fde43429a904ec632a4a91f8f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 42493 } }
I20260926 13:48:45.942229 31666 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8ed181fde43429a904ec632a4a91f8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:48:45.942303 31666 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8ed181fde43429a904ec632a4a91f8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:48:45.942404 31666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8ed181fde43429a904ec632a4a91f8f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:45.943272 31666 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8ed181fde43429a904ec632a4a91f8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ed181fde43429a904ec632a4a91f8f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 42493 } }
I20260926 13:48:45.943616 31666 leader_election.cc:302] T 00000000000000000000000000000000 P a8ed181fde43429a904ec632a4a91f8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8ed181fde43429a904ec632a4a91f8f; no voters:
I20260926 13:48:45.943945 31666 leader_election.cc:288] T 00000000000000000000000000000000 P a8ed181fde43429a904ec632a4a91f8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:48:45.944028 31671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8ed181fde43429a904ec632a4a91f8f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:48:45.944250 31671 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8ed181fde43429a904ec632a4a91f8f [term 1 LEADER]: Becoming Leader. State: Replica: a8ed181fde43429a904ec632a4a91f8f, State: Running, Role: LEADER
I20260926 13:48:45.944537 31671 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8ed181fde43429a904ec632a4a91f8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ed181fde43429a904ec632a4a91f8f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 42493 } }
I20260926 13:48:45.945091 31666 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8ed181fde43429a904ec632a4a91f8f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:48:45.946110 31673 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8ed181fde43429a904ec632a4a91f8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8ed181fde43429a904ec632a4a91f8f. Latest consensus state: current_term: 1 leader_uuid: "a8ed181fde43429a904ec632a4a91f8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ed181fde43429a904ec632a4a91f8f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 42493 } } }
I20260926 13:48:45.946247 31673 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8ed181fde43429a904ec632a4a91f8f [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:45.946496 31672 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8ed181fde43429a904ec632a4a91f8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8ed181fde43429a904ec632a4a91f8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ed181fde43429a904ec632a4a91f8f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 42493 } } }
I20260926 13:48:45.946593 31672 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8ed181fde43429a904ec632a4a91f8f [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:45.946682 31680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:48:45.949590 31680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:48:45.954566 31680 catalog_manager.cc:1347] Generated new cluster ID: 9895a837387045c68ebb83c24c1ce0f2
I20260926 13:48:45.954632 31680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:48:45.973618 31680 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:48:45.974468 31680 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:48:45.990006 31680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8ed181fde43429a904ec632a4a91f8f: Generated new TSK 0
I20260926 13:48:45.990631 31680 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:48:45.995815 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:0
--local_ip_for_outbound_sockets=127.12.170.65
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:42493
--builtin_ntp_servers=127.12.170.84:43073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:48:46.100080 31690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:46.100315 31690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:46.100363 31690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:46.104357 31690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:46.104403 31690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:46.104453 31690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:48:46.108958 31690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:43073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:42493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:46.109862 31690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:46.111088 31690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:46.119822 31696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:46.119860 31698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:46.119860 31695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:46.119915 31690 server_base.cc:1034] running on GCE node
I20260926 13:48:46.120430 31690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:46.120983 31690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:46.122139 31690 hybrid_clock.cc:648] HybridClock initialized: now 1790430526122116 us; error 36 us; skew 500 ppm
I20260926 13:48:46.123876 31690 webserver.cc:460] Webserver started at http://127.12.170.65:40529/ using document root <none> and password file <none>
I20260926 13:48:46.124389 31690 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:46.124451 31690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:46.124622 31690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:46.126514 31690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-0/data/instance:
uuid: "cfdfc10ee289470d86e1288f813f42d2"
format_stamp: "Formatted at 2026-09-26 13:48:46 on dist-test-slave-hc1k"
server_key: "39f7550639025053149daeff166c5ddd"
server_key_iv: "b70d313e54ae223034af26cd53ed927a"
server_key_version: "encryptionkey@0"
I20260926 13:48:46.127018 31690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance:
uuid: "cfdfc10ee289470d86e1288f813f42d2"
format_stamp: "Formatted at 2026-09-26 13:48:46 on dist-test-slave-hc1k"
server_key: "39f7550639025053149daeff166c5ddd"
server_key_iv: "b70d313e54ae223034af26cd53ed927a"
server_key_version: "encryptionkey@0"
I20260926 13:48:46.130432 31690 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260926 13:48:46.132650 31704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:46.133769 31690 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 13:48:46.133874 31690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "cfdfc10ee289470d86e1288f813f42d2"
format_stamp: "Formatted at 2026-09-26 13:48:46 on dist-test-slave-hc1k"
server_key: "39f7550639025053149daeff166c5ddd"
server_key_iv: "b70d313e54ae223034af26cd53ed927a"
server_key_version: "encryptionkey@0"
I20260926 13:48:46.133960 31690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:46.157719 31690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:46.158377 31690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:46.158545 31690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:46.159094 31690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:48:46.160043 31690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:48:46.160090 31690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:46.160141 31690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:48:46.160169 31690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:46.167969 31690 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:40103
I20260926 13:48:46.167987 31817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:40103 every 8 connection(s)
I20260926 13:48:46.168816 31690 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:48:46.171693 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 31690
I20260926 13:48:46.171820 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance
I20260926 13:48:46.172123 12969 external_mini_cluster.cc:1428] Setting key 13dd7f2c13287a793eb784d53c4677f7
I20260926 13:48:46.174217 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:0
--local_ip_for_outbound_sockets=127.12.170.66
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:42493
--builtin_ntp_servers=127.12.170.84:43073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 13:48:46.178049 31669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:46.170488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:59351 (local address 127.12.170.126:42493)
0926 13:48:46.170615 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:46.170619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:46.171727 (+ 1108us) server_negotiation.cc:407] Connection header received
0926 13:48:46.173207 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:46.173210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:46.173255 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:46.173320 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:46.174337 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.174878 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.176171 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.176512 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.176615 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:46.177575 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:46.177579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:46.177582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:48:46.177595 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:46.177619 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:48:46.177629 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:46.177700 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:46.177814 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 13:48:46.177899 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 13:48:46.178789 31820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:46.170867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:42493 (local address 127.12.170.65:59351)
0926 13:48:46.171541 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:46.171595 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:46.173018 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:46.173463 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:46.173470 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:46.173551 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:48:46.174178 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:46.174198 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.175000 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.175003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:46.176054 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:46.176064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.176639 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.176641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:46.176774 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:46.177305 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:48:46.177306 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:48:46.177347 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:48:46.177429 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:48:46.177705 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:48:46.177710 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:48:46.177714 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:48:46.177795 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 13:48:46.177994 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":108,"threads_started":1}
I20260926 13:48:46.179888 31818 heartbeater.cc:347] Connected to a master server at 127.12.170.126:42493
I20260926 13:48:46.180122 31818 heartbeater.cc:464] Registering TS with master...
I20260926 13:48:46.180701 31818 heartbeater.cc:511] Master 127.12.170.126:42493 requested a full tablet report, sending...
I20260926 13:48:46.182139 31630 ts_manager.cc:194] Registered new tserver with Master: cfdfc10ee289470d86e1288f813f42d2 (127.12.170.65:40103)
I20260926 13:48:46.183223 31630 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.170.65:59351
W20260926 13:48:46.288303 31821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:46.288537 31821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:46.288573 31821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:46.292497 31821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:46.292547 31821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:46.292608 31821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:48:46.297269 31821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:43073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:42493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:46.298262 31821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:46.299551 31821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:46.308878 31827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:46.308991 31829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:46.309047 31821 server_base.cc:1034] running on GCE node
W20260926 13:48:46.308912 31826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:46.309464 31821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:46.310014 31821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:46.311172 31821 hybrid_clock.cc:648] HybridClock initialized: now 1790430526311154 us; error 28 us; skew 500 ppm
I20260926 13:48:46.312886 31821 webserver.cc:460] Webserver started at http://127.12.170.66:36829/ using document root <none> and password file <none>
I20260926 13:48:46.313400 31821 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:46.313455 31821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:46.313637 31821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:46.315521 31821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-1/data/instance:
uuid: "30a38f2d616d4c78810117031f97f550"
format_stamp: "Formatted at 2026-09-26 13:48:46 on dist-test-slave-hc1k"
server_key: "21ddea69293d0d2f5cf844b64655575b"
server_key_iv: "9564660ca25a5276e154c50877d583e3"
server_key_version: "encryptionkey@0"
I20260926 13:48:46.316067 31821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance:
uuid: "30a38f2d616d4c78810117031f97f550"
format_stamp: "Formatted at 2026-09-26 13:48:46 on dist-test-slave-hc1k"
server_key: "21ddea69293d0d2f5cf844b64655575b"
server_key_iv: "9564660ca25a5276e154c50877d583e3"
server_key_version: "encryptionkey@0"
I20260926 13:48:46.319358 31821 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:48:46.321587 31835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:46.322753 31821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:48:46.322849 31821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "30a38f2d616d4c78810117031f97f550"
format_stamp: "Formatted at 2026-09-26 13:48:46 on dist-test-slave-hc1k"
server_key: "21ddea69293d0d2f5cf844b64655575b"
server_key_iv: "9564660ca25a5276e154c50877d583e3"
server_key_version: "encryptionkey@0"
I20260926 13:48:46.322933 31821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:46.333699 31821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:46.334283 31821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:46.334435 31821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:46.334935 31821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:48:46.335822 31821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:48:46.335867 31821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:46.335918 31821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:48:46.335945 31821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:46.343902 31821 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:45733
I20260926 13:48:46.343957 31948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:45733 every 8 connection(s)
I20260926 13:48:46.344730 31821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:48:46.349889 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 31821
I20260926 13:48:46.350008 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance
I20260926 13:48:46.350281 12969 external_mini_cluster.cc:1428] Setting key 0bf7c0430317270576d26e9c6c7f7d71
I20260926 13:48:46.352308 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:0
--local_ip_for_outbound_sockets=127.12.170.67
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:42493
--builtin_ntp_servers=127.12.170.84:43073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 13:48:46.354322 31669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:46.346449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:47141 (local address 127.12.170.126:42493)
0926 13:48:46.346574 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:46.346578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:46.347616 (+ 1038us) server_negotiation.cc:407] Connection header received
0926 13:48:46.348957 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:46.348960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:46.349016 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:46.349087 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:46.350088 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.350688 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.352225 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.352496 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.352577 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:46.353856 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:46.353860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:46.353862 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:48:46.353876 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:46.353902 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:48:46.353912 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:46.353985 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:46.354112 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 13:48:46.354176 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:48:46.355084 31951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:46.346881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:42493 (local address 127.12.170.66:47141)
0926 13:48:46.347484 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:46.347518 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:46.348762 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:46.349266 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:46.349273 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:46.349361 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:48:46.349900 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:46.349920 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.350820 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.350823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:46.352098 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:46.352110 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.352653 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.352658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:46.352854 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:46.353549 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:48:46.353551 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:48:46.353606 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:48:46.353688 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:48:46.354000 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:48:46.354006 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:48:46.354010 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:48:46.354097 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 13:48:46.354330 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":102,"threads_started":1}
I20260926 13:48:46.356158 31949 heartbeater.cc:347] Connected to a master server at 127.12.170.126:42493
I20260926 13:48:46.356362 31949 heartbeater.cc:464] Registering TS with master...
I20260926 13:48:46.356833 31949 heartbeater.cc:511] Master 127.12.170.126:42493 requested a full tablet report, sending...
I20260926 13:48:46.357740 31630 ts_manager.cc:194] Registered new tserver with Master: 30a38f2d616d4c78810117031f97f550 (127.12.170.66:45733)
I20260926 13:48:46.358263 31630 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.170.66:47141
W20260926 13:48:46.457455 31952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:46.457701 31952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:46.457752 31952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:46.461687 31952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:46.461736 31952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:46.461807 31952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:48:46.466424 31952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:43073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:42493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:46.467391 31952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:46.468734 31952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:46.477586 31957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:46.477625 31958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:46.477600 31960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:46.477819 31952 server_base.cc:1034] running on GCE node
I20260926 13:48:46.478246 31952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:46.478813 31952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:46.479955 31952 hybrid_clock.cc:648] HybridClock initialized: now 1790430526479928 us; error 45 us; skew 500 ppm
I20260926 13:48:46.481583 31952 webserver.cc:460] Webserver started at http://127.12.170.67:44681/ using document root <none> and password file <none>
I20260926 13:48:46.482120 31952 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:46.482189 31952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:46.482368 31952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:46.484321 31952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-2/data/instance:
uuid: "75af0ab75e704939a05bff62f6a5c288"
format_stamp: "Formatted at 2026-09-26 13:48:46 on dist-test-slave-hc1k"
server_key: "d50d59c6c716217d6358ba5454287b5d"
server_key_iv: "3ae01be8a370f7188a9729c55aba1b9c"
server_key_version: "encryptionkey@0"
I20260926 13:48:46.484826 31952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance:
uuid: "75af0ab75e704939a05bff62f6a5c288"
format_stamp: "Formatted at 2026-09-26 13:48:46 on dist-test-slave-hc1k"
server_key: "d50d59c6c716217d6358ba5454287b5d"
server_key_iv: "3ae01be8a370f7188a9729c55aba1b9c"
server_key_version: "encryptionkey@0"
I20260926 13:48:46.488193 31952 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:48:46.490437 31966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:46.491567 31952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:48:46.491703 31952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "75af0ab75e704939a05bff62f6a5c288"
format_stamp: "Formatted at 2026-09-26 13:48:46 on dist-test-slave-hc1k"
server_key: "d50d59c6c716217d6358ba5454287b5d"
server_key_iv: "3ae01be8a370f7188a9729c55aba1b9c"
server_key_version: "encryptionkey@0"
I20260926 13:48:46.491806 31952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:46.505745 31952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:46.506321 31952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:46.506464 31952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:46.506968 31952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:48:46.507879 31952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:48:46.507927 31952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:46.507964 31952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:48:46.507978 31952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:46.515844 31952 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:34673
I20260926 13:48:46.515901 32079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:34673 every 8 connection(s)
I20260926 13:48:46.516685 31952 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:48:46.518643 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 31952
I20260926 13:48:46.518747 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance
I20260926 13:48:46.519001 12969 external_mini_cluster.cc:1428] Setting key ff2773eced3c0b574972907e7e025177
I20260926 13:48:46.525230 31669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:46.518356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:33879 (local address 127.12.170.126:42493)
0926 13:48:46.518512 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:46.518516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:46.519616 (+ 1100us) server_negotiation.cc:407] Connection header received
0926 13:48:46.521169 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:46.521172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:46.521215 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:46.521281 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:46.522043 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.522538 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.523669 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.523898 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.523976 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:46.524867 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:46.524870 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:46.524872 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:48:46.524883 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:46.524902 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:48:46.524910 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:46.524974 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:46.525073 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 13:48:46.525120 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 13:48:46.526048 32082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:46.518750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:42493 (local address 127.12.170.67:33879)
0926 13:48:46.519464 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:46.519513 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:46.520977 (+ 1464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:46.521408 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:46.521414 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:46.521492 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:48:46.521946 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:46.521957 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.522626 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.522628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:46.523550 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:46.523560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.524010 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.524013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:46.524146 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:46.524652 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:48:46.524653 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:48:46.524690 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:48:46.524748 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:48:46.524980 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:48:46.524985 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:48:46.524988 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:48:46.525068 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 13:48:46.525326 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":109,"threads_started":1}
I20260926 13:48:46.527033 32080 heartbeater.cc:347] Connected to a master server at 127.12.170.126:42493
I20260926 13:48:46.527285 32080 heartbeater.cc:464] Registering TS with master...
I20260926 13:48:46.527812 32080 heartbeater.cc:511] Master 127.12.170.126:42493 requested a full tablet report, sending...
I20260926 13:48:46.528748 31630 ts_manager.cc:194] Registered new tserver with Master: 75af0ab75e704939a05bff62f6a5c288 (127.12.170.67:34673)
I20260926 13:48:46.529285 31630 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.170.67:33879
I20260926 13:48:46.532546 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:48:46.537954 31669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:46.534425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47344 (local address 127.12.170.126:42493)
0926 13:48:46.534516 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:46.534519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:46.534704 (+ 185us) server_negotiation.cc:407] Connection header received
0926 13:48:46.534817 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:46.534819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:46.534851 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:46.534909 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:46.535117 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.535584 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.536480 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.536687 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.536771 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:46.537533 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:46.537535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:46.537537 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:48:46.537548 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:46.537566 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:48:46.537573 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:46.537665 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:46.537821 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 13:48:46.537850 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 13:48:46.540534 31630 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:48:46.543344 31630 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:48:46.556871 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:46.551281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:40103 (local address 127.0.0.1:37840)
0926 13:48:46.551624 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:46.551697 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:46.551828 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:46.552743 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:46.552749 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:46.552773 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:46.552917 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:46.552924 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.554239 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.554242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:46.555425 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:46.555435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.556346 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.556350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:46.556463 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:46.556501 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 13:48:46.556608 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 13:48:46.556715 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":70,"threads_started":1}
I20260926 13:48:46.556954 32092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:46.551422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37840 (local address 127.12.170.65:40103)
0926 13:48:46.552226 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:46.552250 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:46.552279 (+ 29us) server_negotiation.cc:407] Connection header received
0926 13:48:46.552359 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:46.552364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:46.552570 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:46.553064 (+ 494us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:46.553200 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.554154 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.555529 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.556229 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.556334 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:46.556523 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:46.556667 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 13:48:46.556776 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":166,"threads_started":1}
I20260926 13:48:46.557606 32093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:46.551972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:45733 (local address 127.0.0.1:54810)
0926 13:48:46.553437 (+ 1465us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:46.553451 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:46.553544 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:46.553973 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:46.553976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:46.553991 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:46.554067 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:46.554072 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.555151 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.555172 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:46.556227 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:46.556237 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.557205 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.557210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:46.557301 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:46.557328 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 13:48:46.557396 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 13:48:46.557464 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1404,"mutex_wait_us":1,"thread_start_us":70,"threads_started":1}
I20260926 13:48:46.557660 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:46.552160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54810 (local address 127.12.170.66:45733)
0926 13:48:46.552470 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:46.552482 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:46.553553 (+ 1071us) server_negotiation.cc:407] Connection header received
0926 13:48:46.553639 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:46.553644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:46.553833 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:46.553990 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:46.554162 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.555044 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.556361 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.557011 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.557201 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:46.557375 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:46.557426 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 13:48:46.557512 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":111,"threads_started":1}
I20260926 13:48:46.558418 32094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:46.551861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:34673 (local address 127.0.0.1:40522)
0926 13:48:46.552525 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:46.552539 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:46.552616 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:46.553219 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:46.553222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:46.553240 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:46.553320 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:46.553326 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.554659 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.554662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:46.556854 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:46.556863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.558036 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.558040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:46.558125 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:46.558149 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 13:48:46.558219 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 13:48:46.558295 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":1094,"thread_start_us":96,"threads_started":1}
I20260926 13:48:46.558808 32096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:46.552223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40522 (local address 127.12.170.67:34673)
0926 13:48:46.552804 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:46.552814 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:46.552836 (+ 22us) server_negotiation.cc:407] Connection header received
0926 13:48:46.552912 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:46.552916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:46.553091 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:46.553226 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:46.553440 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.554530 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.556983 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.557728 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.557823 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:46.557975 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:46.558573 (+ 598us) server_negotiation.cc:299] Negotiation successful
0926 13:48:46.558662 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":130,"threads_started":1}
I20260926 13:48:46.560109 31752 tablet_service.cc:1467] Processing CreateTablet for tablet 172604918b404a21afc53b5b809177ce (DEFAULT_TABLE table=test-table [id=64ba620f672f4e79a01fd0e9b441514c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:48:46.560604 31883 tablet_service.cc:1467] Processing CreateTablet for tablet 172604918b404a21afc53b5b809177ce (DEFAULT_TABLE table=test-table [id=64ba620f672f4e79a01fd0e9b441514c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:48:46.561151 31752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 172604918b404a21afc53b5b809177ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:46.561443 31883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 172604918b404a21afc53b5b809177ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:46.562239 32014 tablet_service.cc:1467] Processing CreateTablet for tablet 172604918b404a21afc53b5b809177ce (DEFAULT_TABLE table=test-table [id=64ba620f672f4e79a01fd0e9b441514c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:48:46.563175 32014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 172604918b404a21afc53b5b809177ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:46.567087 32097 tablet_bootstrap.cc:492] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550: Bootstrap starting.
I20260926 13:48:46.568385 32098 tablet_bootstrap.cc:492] T 172604918b404a21afc53b5b809177ce P cfdfc10ee289470d86e1288f813f42d2: Bootstrap starting.
I20260926 13:48:46.569697 32099 tablet_bootstrap.cc:492] T 172604918b404a21afc53b5b809177ce P 75af0ab75e704939a05bff62f6a5c288: Bootstrap starting.
I20260926 13:48:46.570416 32097 tablet_bootstrap.cc:654] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:46.570602 32098 tablet_bootstrap.cc:654] T 172604918b404a21afc53b5b809177ce P cfdfc10ee289470d86e1288f813f42d2: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:46.571265 32097 log.cc:824] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:46.571265 32098 log.cc:824] T 172604918b404a21afc53b5b809177ce P cfdfc10ee289470d86e1288f813f42d2: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:46.571468 32099 tablet_bootstrap.cc:654] T 172604918b404a21afc53b5b809177ce P 75af0ab75e704939a05bff62f6a5c288: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:46.572280 32099 log.cc:824] T 172604918b404a21afc53b5b809177ce P 75af0ab75e704939a05bff62f6a5c288: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:46.573232 32097 tablet_bootstrap.cc:492] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550: No bootstrap required, opened a new log
I20260926 13:48:46.573418 32097 ts_tablet_manager.cc:1397] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 13:48:46.573539 32098 tablet_bootstrap.cc:492] T 172604918b404a21afc53b5b809177ce P cfdfc10ee289470d86e1288f813f42d2: No bootstrap required, opened a new log
I20260926 13:48:46.573711 32098 ts_tablet_manager.cc:1397] T 172604918b404a21afc53b5b809177ce P cfdfc10ee289470d86e1288f813f42d2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 13:48:46.574039 32099 tablet_bootstrap.cc:492] T 172604918b404a21afc53b5b809177ce P 75af0ab75e704939a05bff62f6a5c288: No bootstrap required, opened a new log
I20260926 13:48:46.574208 32099 ts_tablet_manager.cc:1397] T 172604918b404a21afc53b5b809177ce P 75af0ab75e704939a05bff62f6a5c288: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 13:48:46.577195 32099 raft_consensus.cc:348] T 172604918b404a21afc53b5b809177ce P 75af0ab75e704939a05bff62f6a5c288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfdfc10ee289470d86e1288f813f42d2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40103 } } peers { permanent_uuid: "30a38f2d616d4c78810117031f97f550" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45733 } } peers { permanent_uuid: "75af0ab75e704939a05bff62f6a5c288" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34673 } }
I20260926 13:48:46.577402 32099 raft_consensus.cc:374] T 172604918b404a21afc53b5b809177ce P 75af0ab75e704939a05bff62f6a5c288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:46.577457 32099 raft_consensus.cc:729] T 172604918b404a21afc53b5b809177ce P 75af0ab75e704939a05bff62f6a5c288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75af0ab75e704939a05bff62f6a5c288, State: Initialized, Role: FOLLOWER
I20260926 13:48:46.577554 32098 raft_consensus.cc:348] T 172604918b404a21afc53b5b809177ce P cfdfc10ee289470d86e1288f813f42d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfdfc10ee289470d86e1288f813f42d2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40103 } } peers { permanent_uuid: "30a38f2d616d4c78810117031f97f550" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45733 } } peers { permanent_uuid: "75af0ab75e704939a05bff62f6a5c288" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34673 } }
I20260926 13:48:46.577553 32097 raft_consensus.cc:348] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfdfc10ee289470d86e1288f813f42d2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40103 } } peers { permanent_uuid: "30a38f2d616d4c78810117031f97f550" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45733 } } peers { permanent_uuid: "75af0ab75e704939a05bff62f6a5c288" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34673 } }
I20260926 13:48:46.577790 32097 raft_consensus.cc:374] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:46.577790 32098 raft_consensus.cc:374] T 172604918b404a21afc53b5b809177ce P cfdfc10ee289470d86e1288f813f42d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:46.577845 32097 raft_consensus.cc:729] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30a38f2d616d4c78810117031f97f550, State: Initialized, Role: FOLLOWER
I20260926 13:48:46.577852 32098 raft_consensus.cc:729] T 172604918b404a21afc53b5b809177ce P cfdfc10ee289470d86e1288f813f42d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfdfc10ee289470d86e1288f813f42d2, State: Initialized, Role: FOLLOWER
I20260926 13:48:46.577886 32099 consensus_queue.cc:260] T 172604918b404a21afc53b5b809177ce P 75af0ab75e704939a05bff62f6a5c288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfdfc10ee289470d86e1288f813f42d2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40103 } } peers { permanent_uuid: "30a38f2d616d4c78810117031f97f550" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45733 } } peers { permanent_uuid: "75af0ab75e704939a05bff62f6a5c288" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34673 } }
I20260926 13:48:46.578289 32098 consensus_queue.cc:260] T 172604918b404a21afc53b5b809177ce P cfdfc10ee289470d86e1288f813f42d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfdfc10ee289470d86e1288f813f42d2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40103 } } peers { permanent_uuid: "30a38f2d616d4c78810117031f97f550" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45733 } } peers { permanent_uuid: "75af0ab75e704939a05bff62f6a5c288" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34673 } }
I20260926 13:48:46.578289 32097 consensus_queue.cc:260] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfdfc10ee289470d86e1288f813f42d2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40103 } } peers { permanent_uuid: "30a38f2d616d4c78810117031f97f550" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45733 } } peers { permanent_uuid: "75af0ab75e704939a05bff62f6a5c288" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34673 } }
I20260926 13:48:46.578874 32099 ts_tablet_manager.cc:1428] T 172604918b404a21afc53b5b809177ce P 75af0ab75e704939a05bff62f6a5c288: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 13:48:46.578948 31949 heartbeater.cc:503] Master 127.12.170.126:42493 was elected leader, sending a full tablet report...
I20260926 13:48:46.579164 32097 ts_tablet_manager.cc:1428] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 13:48:46.579180 32080 heartbeater.cc:503] Master 127.12.170.126:42493 was elected leader, sending a full tablet report...
I20260926 13:48:46.580221 31818 heartbeater.cc:503] Master 127.12.170.126:42493 was elected leader, sending a full tablet report...
I20260926 13:48:46.580533 32098 ts_tablet_manager.cc:1428] T 172604918b404a21afc53b5b809177ce P cfdfc10ee289470d86e1288f813f42d2: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260926 13:48:46.595563 31950 tablet.cc:2369] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:48:46.669889 31819 tablet.cc:2369] T 172604918b404a21afc53b5b809177ce P cfdfc10ee289470d86e1288f813f42d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:48:46.767575 32081 tablet.cc:2369] T 172604918b404a21afc53b5b809177ce P 75af0ab75e704939a05bff62f6a5c288: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:48:46.813872 32104 raft_consensus.cc:482] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:48:46.814083 32104 raft_consensus.cc:504] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfdfc10ee289470d86e1288f813f42d2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40103 } } peers { permanent_uuid: "30a38f2d616d4c78810117031f97f550" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45733 } } peers { permanent_uuid: "75af0ab75e704939a05bff62f6a5c288" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34673 } }
I20260926 13:48:46.815367 32104 leader_election.cc:288] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfdfc10ee289470d86e1288f813f42d2 (127.12.170.65:40103), 75af0ab75e704939a05bff62f6a5c288 (127.12.170.67:34673)
I20260926 13:48:46.818884 31951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:46.815569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:34673 (local address 127.12.170.66:48865)
0926 13:48:46.815778 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:46.815794 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:46.815884 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:46.816153 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:46.816156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:46.816175 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:46.816299 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:46.816307 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.817064 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.817067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:46.817855 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:46.817864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.818497 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.818500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:46.818593 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:46.818616 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 13:48:46.818682 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 13:48:46.818736 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260926 13:48:46.818902 32096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:46.815747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:48865 (local address 127.12.170.67:34673)
0926 13:48:46.815869 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:46.815873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:46.815888 (+ 15us) server_negotiation.cc:407] Connection header received
0926 13:48:46.815999 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:46.816002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:46.816053 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:46.816127 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:46.816403 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.816959 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.817980 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.818385 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.818471 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:46.818508 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:46.818707 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 13:48:46.818783 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 13:48:46.818939 32106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:46.815569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:40103 (local address 127.12.170.66:56855)
0926 13:48:46.816013 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:46.816026 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:46.816134 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:46.816354 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:46.816357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:46.816373 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:46.816485 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:46.816491 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.817168 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.817171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:46.817930 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:46.817941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.818587 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.818589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:46.818659 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:46.818681 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 13:48:46.818758 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 13:48:46.818816 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":28,"thread_start_us":130,"threads_started":1}
I20260926 13:48:46.818957 32092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:46.815684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:56855 (local address 127.12.170.65:40103)
0926 13:48:46.815817 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:46.815822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:46.816125 (+ 303us) server_negotiation.cc:407] Connection header received
0926 13:48:46.816216 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:46.816219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:46.816269 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:46.816371 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:46.816589 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.817069 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.818057 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.818473 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.818580 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:46.818624 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:46.818785 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 13:48:46.818856 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 13:48:46.819541 32034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "172604918b404a21afc53b5b809177ce" candidate_uuid: "30a38f2d616d4c78810117031f97f550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75af0ab75e704939a05bff62f6a5c288" is_pre_election: true
I20260926 13:48:46.819535 31772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "172604918b404a21afc53b5b809177ce" candidate_uuid: "30a38f2d616d4c78810117031f97f550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfdfc10ee289470d86e1288f813f42d2" is_pre_election: true
I20260926 13:48:46.819818 31772 raft_consensus.cc:2454] T 172604918b404a21afc53b5b809177ce P cfdfc10ee289470d86e1288f813f42d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30a38f2d616d4c78810117031f97f550 in term 0.
I20260926 13:48:46.819811 32034 raft_consensus.cc:2454] T 172604918b404a21afc53b5b809177ce P 75af0ab75e704939a05bff62f6a5c288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30a38f2d616d4c78810117031f97f550 in term 0.
I20260926 13:48:46.820223 31838 leader_election.cc:302] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a38f2d616d4c78810117031f97f550, 75af0ab75e704939a05bff62f6a5c288; no voters:
I20260926 13:48:46.820472 32104 raft_consensus.cc:2781] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:48:46.820541 32104 raft_consensus.cc:482] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:48:46.820572 32104 raft_consensus.cc:3037] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:46.821538 32104 raft_consensus.cc:504] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfdfc10ee289470d86e1288f813f42d2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40103 } } peers { permanent_uuid: "30a38f2d616d4c78810117031f97f550" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45733 } } peers { permanent_uuid: "75af0ab75e704939a05bff62f6a5c288" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34673 } }
I20260926 13:48:46.821854 32104 leader_election.cc:288] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550 [CANDIDATE]: Term 1 election: Requested vote from peers cfdfc10ee289470d86e1288f813f42d2 (127.12.170.65:40103), 75af0ab75e704939a05bff62f6a5c288 (127.12.170.67:34673)
I20260926 13:48:46.822170 31772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "172604918b404a21afc53b5b809177ce" candidate_uuid: "30a38f2d616d4c78810117031f97f550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfdfc10ee289470d86e1288f813f42d2"
I20260926 13:48:46.822170 32034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "172604918b404a21afc53b5b809177ce" candidate_uuid: "30a38f2d616d4c78810117031f97f550" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75af0ab75e704939a05bff62f6a5c288"
I20260926 13:48:46.822290 31772 raft_consensus.cc:3037] T 172604918b404a21afc53b5b809177ce P cfdfc10ee289470d86e1288f813f42d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:46.822290 32034 raft_consensus.cc:3037] T 172604918b404a21afc53b5b809177ce P 75af0ab75e704939a05bff62f6a5c288 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:46.823489 32034 raft_consensus.cc:2454] T 172604918b404a21afc53b5b809177ce P 75af0ab75e704939a05bff62f6a5c288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30a38f2d616d4c78810117031f97f550 in term 1.
I20260926 13:48:46.823489 31772 raft_consensus.cc:2454] T 172604918b404a21afc53b5b809177ce P cfdfc10ee289470d86e1288f813f42d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30a38f2d616d4c78810117031f97f550 in term 1.
I20260926 13:48:46.823820 31839 leader_election.cc:302] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30a38f2d616d4c78810117031f97f550, cfdfc10ee289470d86e1288f813f42d2; no voters:
I20260926 13:48:46.823982 32104 raft_consensus.cc:2781] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:48:46.824165 32104 raft_consensus.cc:686] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550 [term 1 LEADER]: Becoming Leader. State: Replica: 30a38f2d616d4c78810117031f97f550, State: Running, Role: LEADER
I20260926 13:48:46.824470 32104 consensus_queue.cc:237] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfdfc10ee289470d86e1288f813f42d2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40103 } } peers { permanent_uuid: "30a38f2d616d4c78810117031f97f550" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45733 } } peers { permanent_uuid: "75af0ab75e704939a05bff62f6a5c288" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34673 } }
I20260926 13:48:46.827323 31630 catalog_manager.cc:5495] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30a38f2d616d4c78810117031f97f550 (127.12.170.66). New cstate: current_term: 1 leader_uuid: "30a38f2d616d4c78810117031f97f550" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfdfc10ee289470d86e1288f813f42d2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30a38f2d616d4c78810117031f97f550" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 45733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75af0ab75e704939a05bff62f6a5c288" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34673 } health_report { overall_health: UNKNOWN } } }
I20260926 13:48:46.852758 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:46.849427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37918 (local address 127.12.170.66:45733)
0926 13:48:46.849536 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:46.849539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:46.849571 (+ 32us) server_negotiation.cc:407] Connection header received
0926 13:48:46.849635 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:46.849638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:46.849690 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:46.849771 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:48:46.849984 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.850499 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.851327 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:46.851548 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:46.851612 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:46.851968 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:48:46.852357 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:48:46.852463 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:46.852608 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 13:48:46.852638 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 13:48:46.858171 31772 raft_consensus.cc:1261] T 172604918b404a21afc53b5b809177ce P cfdfc10ee289470d86e1288f813f42d2 [term 1 FOLLOWER]: Refusing update from remote peer 30a38f2d616d4c78810117031f97f550: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:48:46.858171 32034 raft_consensus.cc:1261] T 172604918b404a21afc53b5b809177ce P 75af0ab75e704939a05bff62f6a5c288 [term 1 FOLLOWER]: Refusing update from remote peer 30a38f2d616d4c78810117031f97f550: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:48:46.858822 32104 consensus_queue.cc:1035] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75af0ab75e704939a05bff62f6a5c288" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 34673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:48:46.858994 32107 consensus_queue.cc:1035] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfdfc10ee289470d86e1288f813f42d2" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:48:46.865114 32110 mvcc.cc:205] Tried to move back new op lower bound from 7333603438004801536 to 7333603437875425280. Current Snapshot: MvccSnapshot[applied={T|T < 7333603438004801536}]
I20260926 13:48:46.866465 32111 mvcc.cc:205] Tried to move back new op lower bound from 7333603438004801536 to 7333603437875425280. Current Snapshot: MvccSnapshot[applied={T|T < 7333603438004801536 or (T in {7333603438004801536})}]
I20260926 13:48:46.869819 32112 mvcc.cc:205] Tried to move back new op lower bound from 7333603438004801536 to 7333603437875425280. Current Snapshot: MvccSnapshot[applied={T|T < 7333603438004801536}]
I20260926 13:48:46.873682 31629 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:48:46.873814 31629 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:48:46.876081 31629 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a8ed181fde43429a904ec632a4a91f8f: Sending DeleteTablet for 3 replicas of tablet 172604918b404a21afc53b5b809177ce
I20260926 13:48:46.876767 32014 tablet_service.cc:1514] Processing DeleteTablet for tablet 172604918b404a21afc53b5b809177ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:48:46 UTC) from {username='slave'} at 127.0.0.1:40522
I20260926 13:48:46.877008 31752 tablet_service.cc:1514] Processing DeleteTablet for tablet 172604918b404a21afc53b5b809177ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:48:46 UTC) from {username='slave'} at 127.0.0.1:37840
I20260926 13:48:46.877188 32120 tablet_replica.cc:330] T 172604918b404a21afc53b5b809177ce P 75af0ab75e704939a05bff62f6a5c288: stopping tablet replica
I20260926 13:48:46.877372 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 31690
I20260926 13:48:46.877475 32120 raft_consensus.cc:2229] T 172604918b404a21afc53b5b809177ce P 75af0ab75e704939a05bff62f6a5c288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:48:46.877480 31883 tablet_service.cc:1514] Processing DeleteTablet for tablet 172604918b404a21afc53b5b809177ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:48:46 UTC) from {username='slave'} at 127.0.0.1:54810
I20260926 13:48:46.877731 32120 raft_consensus.cc:2258] T 172604918b404a21afc53b5b809177ce P 75af0ab75e704939a05bff62f6a5c288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:48:46.879230 32120 ts_tablet_manager.cc:1905] T 172604918b404a21afc53b5b809177ce P 75af0ab75e704939a05bff62f6a5c288: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:48:46.881724 32120 ts_tablet_manager.cc:1918] T 172604918b404a21afc53b5b809177ce P 75af0ab75e704939a05bff62f6a5c288: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:48:46.881798 32120 log.cc:1196] T 172604918b404a21afc53b5b809177ce P 75af0ab75e704939a05bff62f6a5c288: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/172604918b404a21afc53b5b809177ce
I20260926 13:48:46.882040 32120 ts_tablet_manager.cc:1939] T 172604918b404a21afc53b5b809177ce P 75af0ab75e704939a05bff62f6a5c288: Deleting consensus metadata
I20260926 13:48:46.883003 32122 tablet_replica.cc:330] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550: stopping tablet replica
I20260926 13:48:46.883005 31616 catalog_manager.cc:4841] TS 75af0ab75e704939a05bff62f6a5c288 (127.12.170.67:34673): tablet 172604918b404a21afc53b5b809177ce (table test-table [id=64ba620f672f4e79a01fd0e9b441514c]) successfully deleted
I20260926 13:48:46.883261 32122 raft_consensus.cc:2229] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:48:46.883543 32122 raft_consensus.cc:2258] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:48:46.884886 32122 ts_tablet_manager.cc:1905] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:48:46.887339 32122 ts_tablet_manager.cc:1918] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:48:46.887426 32122 log.cc:1196] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790430291939677-12969-0/minicluster-data/ts-1/wal/wals/172604918b404a21afc53b5b809177ce
I20260926 13:48:46.887732 32122 ts_tablet_manager.cc:1939] T 172604918b404a21afc53b5b809177ce P 30a38f2d616d4c78810117031f97f550: Deleting consensus metadata
W20260926 13:48:46.888557 31617 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.170.65:40103 (error 108)
I20260926 13:48:46.888739 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 31821
I20260926 13:48:46.890028 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:46.889717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:40103 (local address 127.0.0.1:38468)
0926 13:48:46.889887 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:46.889948 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.65:40103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260926 13:48:46.890189 31617 catalog_manager.cc:4584] TS cfdfc10ee289470d86e1288f813f42d2 (127.12.170.65:40103): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 172604918b404a21afc53b5b809177ce: Network error: Client connection negotiation failed: client connection to 127.12.170.65:40103: connect: Connection refused (error 111)
I20260926 13:48:46.893726 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 31952
I20260926 13:48:46.895100 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:46.893906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:45733 (local address 127.0.0.1:37920)
0926 13:48:46.894974 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:46.895032 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.66:45733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":996}
I20260926 13:48:46.898432 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 31599
2026-09-26T13:48:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1165 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:48:46 dist-test-slave-hc1k krb5kdc[32126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:48:46 dist-test-slave-hc1k krb5kdc[32126](info): set up 2 sockets
Sep 26 13:48:46 dist-test-slave-hc1k krb5kdc[32126](info): commencing operation
krb5kdc: starting...
W20260926 13:48:47.851227 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:48:47 dist-test-slave-hc1k krb5kdc[32126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:48:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:48:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:48:47.960867 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:38251
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:37939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:38251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:48:48.063868 32139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:48.064111 32139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:48.064149 32139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:48.068133 32139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:48:48.068188 32139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:48.068205 32139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:48.068220 32139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:48:48.068233 32139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:48:48.073408 32139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:37939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:38251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:38251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:48.074393 32139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:48.075350 32139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:48.083244 32144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:48.083297 32145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:48.083297 32147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:48.083444 32139 server_base.cc:1034] running on GCE node
I20260926 13:48:48.084292 32139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:48.085322 32139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:48.086468 32139 hybrid_clock.cc:648] HybridClock initialized: now 1790430528086451 us; error 28 us; skew 500 ppm
Sep 26 13:48:48 dist-test-slave-hc1k krb5kdc[32126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430528, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:48.088896 32139 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:48:48.089671 32139 webserver.cc:460] Webserver started at http://127.12.170.126:33713/ using document root <none> and password file <none>
I20260926 13:48:48.090157 32139 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:48.090216 32139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:48.090399 32139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:48.092347 32139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "b2a61da731e2480780a67f743984088c"
format_stamp: "Formatted at 2026-09-26 13:48:48 on dist-test-slave-hc1k"
server_key: "6a23a4d64394b1219074ebb4a87bdac3"
server_key_iv: "50e08977d7d1a2822f3b5b125ed4f11f"
server_key_version: "encryptionkey@0"
I20260926 13:48:48.092839 32139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "b2a61da731e2480780a67f743984088c"
format_stamp: "Formatted at 2026-09-26 13:48:48 on dist-test-slave-hc1k"
server_key: "6a23a4d64394b1219074ebb4a87bdac3"
server_key_iv: "50e08977d7d1a2822f3b5b125ed4f11f"
server_key_version: "encryptionkey@0"
I20260926 13:48:48.096199 32139 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:48:48.098520 32154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:48.099763 32139 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:48:48.099879 32139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "b2a61da731e2480780a67f743984088c"
format_stamp: "Formatted at 2026-09-26 13:48:48 on dist-test-slave-hc1k"
server_key: "6a23a4d64394b1219074ebb4a87bdac3"
server_key_iv: "50e08977d7d1a2822f3b5b125ed4f11f"
server_key_version: "encryptionkey@0"
I20260926 13:48:48.099967 32139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:48.114689 32139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:48.117384 32139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:48.117538 32139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:48.123787 32139 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:38251
I20260926 13:48:48.123834 32206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:38251 every 8 connection(s)
I20260926 13:48:48.124647 32139 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:48:48.125727 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 32139
I20260926 13:48:48.125852 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:48:48.126106 12969 external_mini_cluster.cc:1428] Setting key 40098efc69be9b0bba5ec19e8251f0e9
I20260926 13:48:48.127418 32207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:48.132526 32207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2a61da731e2480780a67f743984088c: Bootstrap starting.
Sep 26 13:48:48 dist-test-slave-hc1k krb5kdc[32126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:48:48.134951 32207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2a61da731e2480780a67f743984088c: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:48.135773 32207 log.cc:824] T 00000000000000000000000000000000 P b2a61da731e2480780a67f743984088c: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:48.137714 32207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2a61da731e2480780a67f743984088c: No bootstrap required, opened a new log
I20260926 13:48:48.138970 32210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:48.127322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37870 (local address 127.12.170.126:38251)
0926 13:48:48.127706 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:48.127725 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:48.127960 (+ 235us) server_negotiation.cc:407] Connection header received
0926 13:48:48.128653 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:48.128669 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:48.128974 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:48.129254 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:48.130026 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.130975 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.131867 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.132316 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.132454 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:48.134883 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:48.134888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:48.134900 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:48.134925 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:48.136435 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:48.136884 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:48.136888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:48.136892 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:48.136996 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:48.137257 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:48.137259 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:48.137271 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:48.137617 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:48.137762 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:48.138041 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 13:48:48.138308 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":124,"threads_started":1}
I20260926 13:48:48.141113 32207 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2a61da731e2480780a67f743984088c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a61da731e2480780a67f743984088c" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 38251 } }
I20260926 13:48:48.141319 32207 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2a61da731e2480780a67f743984088c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:48.141374 32207 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2a61da731e2480780a67f743984088c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a61da731e2480780a67f743984088c, State: Initialized, Role: FOLLOWER
I20260926 13:48:48.141772 32207 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2a61da731e2480780a67f743984088c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a61da731e2480780a67f743984088c" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 38251 } }
I20260926 13:48:48.141888 32207 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2a61da731e2480780a67f743984088c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:48:48.141947 32207 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2a61da731e2480780a67f743984088c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:48:48.142035 32207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2a61da731e2480780a67f743984088c [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:48.142952 32207 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2a61da731e2480780a67f743984088c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a61da731e2480780a67f743984088c" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 38251 } }
I20260926 13:48:48.143293 32207 leader_election.cc:302] T 00000000000000000000000000000000 P b2a61da731e2480780a67f743984088c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2a61da731e2480780a67f743984088c; no voters:
I20260926 13:48:48.143575 32207 leader_election.cc:288] T 00000000000000000000000000000000 P b2a61da731e2480780a67f743984088c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:48:48.143687 32212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2a61da731e2480780a67f743984088c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:48:48.143898 32212 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2a61da731e2480780a67f743984088c [term 1 LEADER]: Becoming Leader. State: Replica: b2a61da731e2480780a67f743984088c, State: Running, Role: LEADER
I20260926 13:48:48.144214 32212 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2a61da731e2480780a67f743984088c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a61da731e2480780a67f743984088c" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 38251 } }
I20260926 13:48:48.144685 32207 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2a61da731e2480780a67f743984088c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:48:48.145831 32214 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2a61da731e2480780a67f743984088c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2a61da731e2480780a67f743984088c. Latest consensus state: current_term: 1 leader_uuid: "b2a61da731e2480780a67f743984088c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a61da731e2480780a67f743984088c" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 38251 } } }
I20260926 13:48:48.145859 32213 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2a61da731e2480780a67f743984088c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2a61da731e2480780a67f743984088c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a61da731e2480780a67f743984088c" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 38251 } } }
I20260926 13:48:48.145998 32214 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2a61da731e2480780a67f743984088c [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:48.146034 32213 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2a61da731e2480780a67f743984088c [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:48.146739 32221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:48:48.149257 32221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:48:48.154331 32221 catalog_manager.cc:1347] Generated new cluster ID: 45bfd949586f4785abea4f335c3a5b7f
I20260926 13:48:48.154390 32221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:48:48.172458 32221 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:48:48.173676 32221 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:48:48.185055 32221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2a61da731e2480780a67f743984088c: Generated new TSK 0
I20260926 13:48:48.185735 32221 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.170.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
Entry for principal kudu/127.12.170.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab.
I20260926 13:48:48.216605 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:0
--local_ip_for_outbound_sockets=127.12.170.65
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:38251
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:37939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:48:48.320924 32233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:48.321178 32233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:48.321223 32233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:48.325428 32233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:48.325479 32233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:48.325533 32233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:48:48.330614 32233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:37939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.65.keytab
--principal=kudu/127.12.170.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:38251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:48.331490 32233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:48.332443 32233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:48.341207 32238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:48.341177 32239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:48.341167 32241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:48.341476 32233 server_base.cc:1034] running on GCE node
I20260926 13:48:48.341846 32233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:48.342378 32233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:48.343529 32233 hybrid_clock.cc:648] HybridClock initialized: now 1790430528343513 us; error 31 us; skew 500 ppm
Sep 26 13:48:48 dist-test-slave-hc1k krb5kdc[32126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430528, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:48.346091 32233 init.cc:377] Logged in from keytab as kudu/127.12.170.65@KRBTEST.COM (short username kudu)
I20260926 13:48:48.346916 32233 webserver.cc:460] Webserver started at http://127.12.170.65:33781/ using document root <none> and password file <none>
I20260926 13:48:48.347422 32233 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:48.347482 32233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:48.347703 32233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:48.349614 32233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-0/data/instance:
uuid: "b573d7b06bff4419b56744f4599681db"
format_stamp: "Formatted at 2026-09-26 13:48:48 on dist-test-slave-hc1k"
server_key: "ccddb8adee0aca9e42fef617b7ffc7dc"
server_key_iv: "88df3403646896ff3c0f5269fd82a79d"
server_key_version: "encryptionkey@0"
I20260926 13:48:48.350132 32233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance:
uuid: "b573d7b06bff4419b56744f4599681db"
format_stamp: "Formatted at 2026-09-26 13:48:48 on dist-test-slave-hc1k"
server_key: "ccddb8adee0aca9e42fef617b7ffc7dc"
server_key_iv: "88df3403646896ff3c0f5269fd82a79d"
server_key_version: "encryptionkey@0"
I20260926 13:48:48.353538 32233 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 13:48:48.355643 32248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:48.356841 32233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:48:48.356940 32233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "b573d7b06bff4419b56744f4599681db"
format_stamp: "Formatted at 2026-09-26 13:48:48 on dist-test-slave-hc1k"
server_key: "ccddb8adee0aca9e42fef617b7ffc7dc"
server_key_iv: "88df3403646896ff3c0f5269fd82a79d"
server_key_version: "encryptionkey@0"
I20260926 13:48:48.357028 32233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:48.384698 32233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:48.387799 32233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:48.387966 32233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:48.388523 32233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:48:48.389314 32233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:48:48.389359 32233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:48.389410 32233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:48:48.389437 32233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:48.397523 32233 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:46509
I20260926 13:48:48.397612 32361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:46509 every 8 connection(s)
I20260926 13:48:48.398365 32233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:48:48.402349 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 32233
I20260926 13:48:48.402457 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance
I20260926 13:48:48.402745 12969 external_mini_cluster.cc:1428] Setting key e6f79287c420e0b468d4dc3d9dd5edf6
Sep 26 13:48:48 dist-test-slave-hc1k krb5kdc[32126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430528, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.65@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.170.66@KRBTEST.COM; defaulting to no policy
I20260926 13:48:48.412058 32210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:48.400135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:47609 (local address 127.12.170.126:38251)
0926 13:48:48.400265 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:48.400269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:48.401107 (+ 838us) server_negotiation.cc:407] Connection header received
0926 13:48:48.402463 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:48.402467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:48.402526 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:48.402611 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:48.403994 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.404704 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.405911 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.406144 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.406239 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:48.408634 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:48.408637 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:48.408639 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:48.408664 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:48.410134 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:48.410620 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:48.410623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:48.410625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:48.410699 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:48.411005 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:48.411008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:48.411010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:48.411191 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:48.411273 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:48.411745 (+ 472us) server_negotiation.cc:299] Negotiation successful
0926 13:48:48.411868 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 13:48:48.412757 32364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:48.400508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:38251 (local address 127.12.170.65:47609)
0926 13:48:48.400938 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:48.400982 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:48.402253 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:48.402789 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:48.402796 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:48.403211 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:48:48.403849 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:48.403871 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.404825 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.404827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:48.405773 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:48.405784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.406284 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.406288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:48.406449 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:48.406994 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:48:48.407011 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:48:48.408475 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:48:48.410280 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:48.410286 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:48.410299 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:48.410529 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:48.410787 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:48.410790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:48.410792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:48.410922 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:48.411288 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:48:48.411294 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:48:48.411511 (+ 217us) client_negotiation.cc:769] Sending connection context
0926 13:48:48.411680 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 13:48:48.411976 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":92,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.66@KRBTEST.COM" created.
I20260926 13:48:48.413869 32362 heartbeater.cc:347] Connected to a master server at 127.12.170.126:38251
I20260926 13:48:48.414078 32362 heartbeater.cc:464] Registering TS with master...
I20260926 13:48:48.414516 32362 heartbeater.cc:511] Master 127.12.170.126:38251 requested a full tablet report, sending...
I20260926 13:48:48.415889 32171 ts_manager.cc:194] Registered new tserver with Master: b573d7b06bff4419b56744f4599681db (127.12.170.65:46509)
I20260926 13:48:48.416962 32171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.65@KRBTEST.COM'} at 127.12.170.65:47609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
Entry for principal kudu/127.12.170.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab.
I20260926 13:48:48.426327 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:0
--local_ip_for_outbound_sockets=127.12.170.66
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:38251
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:37939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:48:48.529049 32367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:48.529286 32367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:48.529343 32367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:48.533183 32367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:48.533231 32367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:48.533313 32367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:48:48.538350 32367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:37939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.66.keytab
--principal=kudu/127.12.170.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:38251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:48.539297 32367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:48.540241 32367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:48.549338 32373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:48.549384 32372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:48.549598 32375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:48.549746 32367 server_base.cc:1034] running on GCE node
I20260926 13:48:48.550103 32367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:48.550707 32367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:48.551847 32367 hybrid_clock.cc:648] HybridClock initialized: now 1790430528551828 us; error 29 us; skew 500 ppm
Sep 26 13:48:48 dist-test-slave-hc1k krb5kdc[32126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430528, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:48.554297 32367 init.cc:377] Logged in from keytab as kudu/127.12.170.66@KRBTEST.COM (short username kudu)
I20260926 13:48:48.555094 32367 webserver.cc:460] Webserver started at http://127.12.170.66:34079/ using document root <none> and password file <none>
I20260926 13:48:48.555598 32367 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:48.555681 32367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:48.555876 32367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:48.557783 32367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-1/data/instance:
uuid: "4a91f7c8f8a04e18ab7fcb4a46a49787"
format_stamp: "Formatted at 2026-09-26 13:48:48 on dist-test-slave-hc1k"
server_key: "8dc83bfbf33f86b330211c465b8769c8"
server_key_iv: "0d5e73e092202f05bed08f522d442789"
server_key_version: "encryptionkey@0"
I20260926 13:48:48.558302 32367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance:
uuid: "4a91f7c8f8a04e18ab7fcb4a46a49787"
format_stamp: "Formatted at 2026-09-26 13:48:48 on dist-test-slave-hc1k"
server_key: "8dc83bfbf33f86b330211c465b8769c8"
server_key_iv: "0d5e73e092202f05bed08f522d442789"
server_key_version: "encryptionkey@0"
I20260926 13:48:48.561748 32367 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:48:48.564000 32382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:48.565158 32367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 13:48:48.565256 32367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "4a91f7c8f8a04e18ab7fcb4a46a49787"
format_stamp: "Formatted at 2026-09-26 13:48:48 on dist-test-slave-hc1k"
server_key: "8dc83bfbf33f86b330211c465b8769c8"
server_key_iv: "0d5e73e092202f05bed08f522d442789"
server_key_version: "encryptionkey@0"
I20260926 13:48:48.565356 32367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:48.598075 32367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:48.600925 32367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:48.601095 32367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:48.601658 32367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:48:48.602445 32367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:48:48.602491 32367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:48.602542 32367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:48:48.602569 32367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:48.610699 32367 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:42809
I20260926 13:48:48.610733 32495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:42809 every 8 connection(s)
I20260926 13:48:48.611539 32367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
Sep 26 13:48:48 dist-test-slave-hc1k krb5kdc[32126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430528, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.66@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:48:48.621608 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 32367
I20260926 13:48:48.621728 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance
I20260926 13:48:48.621974 12969 external_mini_cluster.cc:1428] Setting key a7e211d1d915ac991a0b366c71ad43e2
I20260926 13:48:48.624121 32210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:48.613272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:33413 (local address 127.12.170.126:38251)
0926 13:48:48.613410 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:48.613413 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:48.614153 (+ 740us) server_negotiation.cc:407] Connection header received
0926 13:48:48.615423 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:48.615427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:48.615480 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:48.615551 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:48.616688 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.617204 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.618302 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.618516 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.618603 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:48.620782 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:48.620785 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:48.620788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:48.620811 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:48.622177 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:48.622636 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:48.622640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:48.622642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:48.622724 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:48.623059 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:48.623062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:48.623064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:48.623238 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:48.623341 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:48.623854 (+ 513us) server_negotiation.cc:299] Negotiation successful
0926 13:48:48.623971 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 13:48:48.624826 32498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:48.613604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:38251 (local address 127.12.170.66:33413)
0926 13:48:48.614017 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:48.614058 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:48.615222 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:48.615756 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:48.615771 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:48.616102 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:48:48.616551 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:48.616564 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.617319 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.617322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:48.618184 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:48.618193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.618646 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.618650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:48.618815 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:48.619279 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:48:48.619294 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:48:48.620634 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:48:48.622289 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:48.622295 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:48.622306 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:48.622532 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:48.622802 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:48.622805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:48.622807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:48.622969 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:48.623353 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:48:48.623358 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:48:48.623605 (+ 247us) client_negotiation.cc:769] Sending connection context
0926 13:48:48.623804 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 13:48:48.624089 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":109,"threads_started":1}
I20260926 13:48:48.626335 32496 heartbeater.cc:347] Connected to a master server at 127.12.170.126:38251
I20260926 13:48:48.626541 32496 heartbeater.cc:464] Registering TS with master...
I20260926 13:48:48.626960 32496 heartbeater.cc:511] Master 127.12.170.126:38251 requested a full tablet report, sending...
I20260926 13:48:48.627887 32171 ts_manager.cc:194] Registered new tserver with Master: 4a91f7c8f8a04e18ab7fcb4a46a49787 (127.12.170.66:42809)
I20260926 13:48:48.628470 32171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.66@KRBTEST.COM'} at 127.12.170.66:33413
WARNING: no policy specified for kudu/127.12.170.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
Entry for principal kudu/127.12.170.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab.
I20260926 13:48:48.646620 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:0
--local_ip_for_outbound_sockets=127.12.170.67
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:38251
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:37939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:48:48.751616 32501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:48.751873 32501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:48.751914 32501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:48.755815 32501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:48.755862 32501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:48.755926 32501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:48:48.761122 32501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:37939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.67.keytab
--principal=kudu/127.12.170.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:38251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:48.762089 32501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:48.763036 32501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:48.771920 32509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:48.772027 32506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:48.771996 32501 server_base.cc:1034] running on GCE node
W20260926 13:48:48.772176 32507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:48.772642 32501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:48.773223 32501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:48.774399 32501 hybrid_clock.cc:648] HybridClock initialized: now 1790430528774369 us; error 37 us; skew 500 ppm
Sep 26 13:48:48 dist-test-slave-hc1k krb5kdc[32126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430528, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:48.776893 32501 init.cc:377] Logged in from keytab as kudu/127.12.170.67@KRBTEST.COM (short username kudu)
I20260926 13:48:48.777664 32501 webserver.cc:460] Webserver started at http://127.12.170.67:43937/ using document root <none> and password file <none>
I20260926 13:48:48.778157 32501 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:48.778218 32501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:48.778390 32501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:48.780336 32501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-2/data/instance:
uuid: "8163c130e4c843cab2076141593eae56"
format_stamp: "Formatted at 2026-09-26 13:48:48 on dist-test-slave-hc1k"
server_key: "74886d7dfdbbc35bda1a2991d07a4006"
server_key_iv: "78ca4794b12a9c9b5f78e77e2e7e9db3"
server_key_version: "encryptionkey@0"
I20260926 13:48:48.780839 32501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance:
uuid: "8163c130e4c843cab2076141593eae56"
format_stamp: "Formatted at 2026-09-26 13:48:48 on dist-test-slave-hc1k"
server_key: "74886d7dfdbbc35bda1a2991d07a4006"
server_key_iv: "78ca4794b12a9c9b5f78e77e2e7e9db3"
server_key_version: "encryptionkey@0"
I20260926 13:48:48.784183 32501 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 13:48:48.786348 32516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:48.787408 32501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:48:48.787513 32501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "8163c130e4c843cab2076141593eae56"
format_stamp: "Formatted at 2026-09-26 13:48:48 on dist-test-slave-hc1k"
server_key: "74886d7dfdbbc35bda1a2991d07a4006"
server_key_iv: "78ca4794b12a9c9b5f78e77e2e7e9db3"
server_key_version: "encryptionkey@0"
I20260926 13:48:48.787592 32501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:48.802989 32501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:48.805900 32501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:48.806063 32501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:48.806574 32501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:48:48.807323 32501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:48:48.807366 32501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:48.807416 32501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:48:48.807444 32501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:48.815531 32501 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:42103
I20260926 13:48:48.815558 32629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:42103 every 8 connection(s)
I20260926 13:48:48.816366 32501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:48:48.821660 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 32501
I20260926 13:48:48.821771 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance
I20260926 13:48:48.822017 12969 external_mini_cluster.cc:1428] Setting key 5ea24757d791e971f03003bbfa506a2c
Sep 26 13:48:48 dist-test-slave-hc1k krb5kdc[32126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430528, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.67@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:48:48.828462 32210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:48.817998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:43601 (local address 127.12.170.126:38251)
0926 13:48:48.818115 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:48.818118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:48.818855 (+ 737us) server_negotiation.cc:407] Connection header received
0926 13:48:48.820144 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:48.820147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:48.820229 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:48.820302 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:48.821390 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.821956 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.823050 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.823271 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.823355 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:48.825525 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:48.825528 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:48.825530 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:48.825551 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:48.826632 (+ 1081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:48.827069 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:48.827073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:48.827075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:48.827144 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:48.827449 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:48.827452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:48.827453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:48.827674 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:48.827761 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:48.828196 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 13:48:48.828313 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 13:48:48.829165 32632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:48.818302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:38251 (local address 127.12.170.67:43601)
0926 13:48:48.818716 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:48.818756 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:48.819951 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:48.820475 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:48.820481 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:48.820811 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:48:48.821255 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:48.821268 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.822062 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.822065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:48.822935 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:48.822945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.823396 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.823399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:48.823549 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:48.824060 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:48:48.824075 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:48:48.825380 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:48:48.826738 (+ 1358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:48.826744 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:48.826757 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:48.826982 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:48.827221 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:48:48.827224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:48:48.827226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:48:48.827347 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:48:48.827759 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:48:48.827765 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:48:48.827985 (+ 220us) client_negotiation.cc:769] Sending connection context
0926 13:48:48.828141 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 13:48:48.828433 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":104,"threads_started":1}
I20260926 13:48:48.830083 32630 heartbeater.cc:347] Connected to a master server at 127.12.170.126:38251
I20260926 13:48:48.830278 32630 heartbeater.cc:464] Registering TS with master...
I20260926 13:48:48.830684 32630 heartbeater.cc:511] Master 127.12.170.126:38251 requested a full tablet report, sending...
I20260926 13:48:48.831632 32171 ts_manager.cc:194] Registered new tserver with Master: 8163c130e4c843cab2076141593eae56 (127.12.170.67:42103)
I20260926 13:48:48.832172 32171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.170.67@KRBTEST.COM'} at 127.12.170.67:43601
I20260926 13:48:48.835261 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:48:48.844024 32210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:48.837108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37874 (local address 127.12.170.126:38251)
0926 13:48:48.837218 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:48.837222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:48.837392 (+ 170us) server_negotiation.cc:407] Connection header received
0926 13:48:48.837501 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:48.837504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:48.837545 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:48.837604 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:48.838244 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.838740 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.839544 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.839804 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.839884 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:48.841069 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:48.841073 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:48.841075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:48.841100 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:48.842231 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:48.842688 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:48.842691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:48.842693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:48.842804 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:48.843092 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:48.843095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:48.843096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:48.843365 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:48.843492 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:48.843775 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 13:48:48.843878 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 13:48:48.846733 32171 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:48:48.849725 32171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:48:48.863756 32641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:48.857407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:46509 (local address 127.0.0.1:33274)
0926 13:48:48.857854 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:48.857899 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:48.858028 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:48.859603 (+ 1575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:48.859606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:48.859621 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:48.859750 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:48.859757 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.860865 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.860868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:48.862251 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:48.862262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.863335 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.863339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:48.863420 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:48.863441 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 13:48:48.863511 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 13:48:48.863563 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":59,"thread_start_us":73,"threads_started":1}
I20260926 13:48:48.863570 32643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:48.857567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:42103 (local address 127.0.0.1:44266)
0926 13:48:48.858425 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:48.858446 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:48.858533 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:48.859027 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:48.859032 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:48.859053 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:48.859183 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:48.859190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.861305 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.861307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:48.862256 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:48.862267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.863056 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.863059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:48.863164 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:48.863200 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 13:48:48.863307 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 13:48:48.863420 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"mutex_wait_us":107,"thread_start_us":91,"threads_started":1}
I20260926 13:48:48.863873 32644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:48.857995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44266 (local address 127.12.170.67:42103)
0926 13:48:48.858315 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:48.858326 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:48.858627 (+ 301us) server_negotiation.cc:407] Connection header received
0926 13:48:48.858707 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:48.858713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:48.858891 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:48.859006 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:48.859307 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.860531 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.862391 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.862946 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.863041 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:48.863181 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:48.863635 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 13:48:48.863752 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":156,"threads_started":1}
I20260926 13:48:48.864123 32645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:48.858101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:42809 (local address 127.0.0.1:49286)
0926 13:48:48.858500 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:48.858512 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:48.858594 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:48.859165 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:48.859168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:48.859186 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:48.859288 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:48.859295 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.860650 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.860653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:48.862263 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:48.862269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.863518 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.863522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:48.863620 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:48.863642 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 13:48:48.863731 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 13:48:48.863798 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":116,"thread_start_us":87,"threads_started":1}
I20260926 13:48:48.864315 32642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:48.857572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33274 (local address 127.12.170.65:46509)
0926 13:48:48.859247 (+ 1675us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:48.859261 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:48.859281 (+ 20us) server_negotiation.cc:407] Connection header received
0926 13:48:48.859348 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:48.859353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:48.859493 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:48.859608 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:48.859866 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.860737 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.862390 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.863223 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.863324 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:48.863454 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:48.863538 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 13:48:48.863632 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1572,"thread_start_us":59,"threads_started":1}
I20260926 13:48:48.864816 32646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:48.858359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49286 (local address 127.12.170.66:42809)
0926 13:48:48.858759 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:48.858770 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:48.858792 (+ 22us) server_negotiation.cc:407] Connection header received
0926 13:48:48.858859 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:48.858862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:48.859026 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:48.859169 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:48.859394 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.860522 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.862390 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:48.863223 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:48.863324 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:48.863454 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:48.864477 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0926 13:48:48.864688 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":101,"threads_started":1}
I20260926 13:48:48.867035 32430 tablet_service.cc:1467] Processing CreateTablet for tablet 7ee5cd2088af4a5abd9473873ee95f2c (DEFAULT_TABLE table=test-table [id=0af095d1f7714a5299d1292d5d2af9b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:48:48.867141 32564 tablet_service.cc:1467] Processing CreateTablet for tablet 7ee5cd2088af4a5abd9473873ee95f2c (DEFAULT_TABLE table=test-table [id=0af095d1f7714a5299d1292d5d2af9b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:48:48.867143 32296 tablet_service.cc:1467] Processing CreateTablet for tablet 7ee5cd2088af4a5abd9473873ee95f2c (DEFAULT_TABLE table=test-table [id=0af095d1f7714a5299d1292d5d2af9b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:48:48.868054 32430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ee5cd2088af4a5abd9473873ee95f2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:48.868142 32296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ee5cd2088af4a5abd9473873ee95f2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:48.868142 32564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ee5cd2088af4a5abd9473873ee95f2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:48.873989 32647 tablet_bootstrap.cc:492] T 7ee5cd2088af4a5abd9473873ee95f2c P 4a91f7c8f8a04e18ab7fcb4a46a49787: Bootstrap starting.
I20260926 13:48:48.875294 32648 tablet_bootstrap.cc:492] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56: Bootstrap starting.
I20260926 13:48:48.875967 32649 tablet_bootstrap.cc:492] T 7ee5cd2088af4a5abd9473873ee95f2c P b573d7b06bff4419b56744f4599681db: Bootstrap starting.
I20260926 13:48:48.876760 32647 tablet_bootstrap.cc:654] T 7ee5cd2088af4a5abd9473873ee95f2c P 4a91f7c8f8a04e18ab7fcb4a46a49787: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:48.877318 32648 tablet_bootstrap.cc:654] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:48.877789 32647 log.cc:824] T 7ee5cd2088af4a5abd9473873ee95f2c P 4a91f7c8f8a04e18ab7fcb4a46a49787: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:48.877913 32649 tablet_bootstrap.cc:654] T 7ee5cd2088af4a5abd9473873ee95f2c P b573d7b06bff4419b56744f4599681db: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:48.878528 32648 log.cc:824] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:48.879040 32649 log.cc:824] T 7ee5cd2088af4a5abd9473873ee95f2c P b573d7b06bff4419b56744f4599681db: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:48.879881 32647 tablet_bootstrap.cc:492] T 7ee5cd2088af4a5abd9473873ee95f2c P 4a91f7c8f8a04e18ab7fcb4a46a49787: No bootstrap required, opened a new log
I20260926 13:48:48.880053 32647 ts_tablet_manager.cc:1397] T 7ee5cd2088af4a5abd9473873ee95f2c P 4a91f7c8f8a04e18ab7fcb4a46a49787: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 13:48:48.880393 32648 tablet_bootstrap.cc:492] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56: No bootstrap required, opened a new log
I20260926 13:48:48.880566 32648 ts_tablet_manager.cc:1397] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 13:48:48.880966 32649 tablet_bootstrap.cc:492] T 7ee5cd2088af4a5abd9473873ee95f2c P b573d7b06bff4419b56744f4599681db: No bootstrap required, opened a new log
I20260926 13:48:48.881127 32649 ts_tablet_manager.cc:1397] T 7ee5cd2088af4a5abd9473873ee95f2c P b573d7b06bff4419b56744f4599681db: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 13:48:48.882900 32647 raft_consensus.cc:348] T 7ee5cd2088af4a5abd9473873ee95f2c P 4a91f7c8f8a04e18ab7fcb4a46a49787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b573d7b06bff4419b56744f4599681db" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 46509 } } peers { permanent_uuid: "8163c130e4c843cab2076141593eae56" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42103 } } peers { permanent_uuid: "4a91f7c8f8a04e18ab7fcb4a46a49787" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42809 } }
I20260926 13:48:48.883098 32647 raft_consensus.cc:374] T 7ee5cd2088af4a5abd9473873ee95f2c P 4a91f7c8f8a04e18ab7fcb4a46a49787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:48.883164 32647 raft_consensus.cc:729] T 7ee5cd2088af4a5abd9473873ee95f2c P 4a91f7c8f8a04e18ab7fcb4a46a49787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a91f7c8f8a04e18ab7fcb4a46a49787, State: Initialized, Role: FOLLOWER
I20260926 13:48:48.883577 32647 consensus_queue.cc:260] T 7ee5cd2088af4a5abd9473873ee95f2c P 4a91f7c8f8a04e18ab7fcb4a46a49787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b573d7b06bff4419b56744f4599681db" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 46509 } } peers { permanent_uuid: "8163c130e4c843cab2076141593eae56" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42103 } } peers { permanent_uuid: "4a91f7c8f8a04e18ab7fcb4a46a49787" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42809 } }
I20260926 13:48:48.884361 32647 ts_tablet_manager.cc:1428] T 7ee5cd2088af4a5abd9473873ee95f2c P 4a91f7c8f8a04e18ab7fcb4a46a49787: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 13:48:48.884210 32648 raft_consensus.cc:348] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b573d7b06bff4419b56744f4599681db" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 46509 } } peers { permanent_uuid: "8163c130e4c843cab2076141593eae56" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42103 } } peers { permanent_uuid: "4a91f7c8f8a04e18ab7fcb4a46a49787" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42809 } }
I20260926 13:48:48.884433 32648 raft_consensus.cc:374] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:48.884531 32648 raft_consensus.cc:729] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8163c130e4c843cab2076141593eae56, State: Initialized, Role: FOLLOWER
I20260926 13:48:48.884672 32496 heartbeater.cc:503] Master 127.12.170.126:38251 was elected leader, sending a full tablet report...
I20260926 13:48:48.884639 32649 raft_consensus.cc:348] T 7ee5cd2088af4a5abd9473873ee95f2c P b573d7b06bff4419b56744f4599681db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b573d7b06bff4419b56744f4599681db" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 46509 } } peers { permanent_uuid: "8163c130e4c843cab2076141593eae56" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42103 } } peers { permanent_uuid: "4a91f7c8f8a04e18ab7fcb4a46a49787" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42809 } }
I20260926 13:48:48.884838 32649 raft_consensus.cc:374] T 7ee5cd2088af4a5abd9473873ee95f2c P b573d7b06bff4419b56744f4599681db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:48.884881 32649 raft_consensus.cc:729] T 7ee5cd2088af4a5abd9473873ee95f2c P b573d7b06bff4419b56744f4599681db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b573d7b06bff4419b56744f4599681db, State: Initialized, Role: FOLLOWER
I20260926 13:48:48.885042 32648 consensus_queue.cc:260] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b573d7b06bff4419b56744f4599681db" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 46509 } } peers { permanent_uuid: "8163c130e4c843cab2076141593eae56" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42103 } } peers { permanent_uuid: "4a91f7c8f8a04e18ab7fcb4a46a49787" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42809 } }
I20260926 13:48:48.885282 32649 consensus_queue.cc:260] T 7ee5cd2088af4a5abd9473873ee95f2c P b573d7b06bff4419b56744f4599681db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b573d7b06bff4419b56744f4599681db" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 46509 } } peers { permanent_uuid: "8163c130e4c843cab2076141593eae56" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42103 } } peers { permanent_uuid: "4a91f7c8f8a04e18ab7fcb4a46a49787" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42809 } }
I20260926 13:48:48.885895 32362 heartbeater.cc:503] Master 127.12.170.126:38251 was elected leader, sending a full tablet report...
I20260926 13:48:48.885908 32648 ts_tablet_manager.cc:1428] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 13:48:48.886195 32630 heartbeater.cc:503] Master 127.12.170.126:38251 was elected leader, sending a full tablet report...
I20260926 13:48:48.886813 32649 ts_tablet_manager.cc:1428] T 7ee5cd2088af4a5abd9473873ee95f2c P b573d7b06bff4419b56744f4599681db: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260926 13:48:48.899328 32363 tablet.cc:2369] T 7ee5cd2088af4a5abd9473873ee95f2c P b573d7b06bff4419b56744f4599681db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:48:49.061573 32654 raft_consensus.cc:482] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:48:49.061782 32654 raft_consensus.cc:504] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b573d7b06bff4419b56744f4599681db" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 46509 } } peers { permanent_uuid: "8163c130e4c843cab2076141593eae56" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42103 } } peers { permanent_uuid: "4a91f7c8f8a04e18ab7fcb4a46a49787" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42809 } }
I20260926 13:48:49.062920 32654 leader_election.cc:288] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b573d7b06bff4419b56744f4599681db (127.12.170.65:46509), 4a91f7c8f8a04e18ab7fcb4a46a49787 (127.12.170.66:42809)
I20260926 13:48:49.066365 32632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:49.063165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:46509 (local address 127.12.170.67:56299)
0926 13:48:49.063289 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:49.063305 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:49.063399 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:49.063711 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:49.063714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:49.063734 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:49.063854 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:49.063860 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.064566 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.064570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:49.065366 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:49.065374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.065963 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.065967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:49.066071 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:49.066088 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 13:48:49.066149 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 13:48:49.066215 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"spinlock_wait_cycles":19200}
I20260926 13:48:49.066354 32642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:49.063269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:56299 (local address 127.12.170.65:46509)
0926 13:48:49.063398 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:49.063403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:49.063418 (+ 15us) server_negotiation.cc:407] Connection header received
0926 13:48:49.063529 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:49.063532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:49.063586 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:49.063691 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:49.063959 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.064469 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.065482 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.065866 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.065964 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:49.066017 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:49.066171 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 13:48:49.066240 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 13:48:49.066604 32646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:49.063273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:39831 (local address 127.12.170.66:42809)
0926 13:48:49.063398 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:49.063403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:49.063703 (+ 300us) server_negotiation.cc:407] Connection header received
0926 13:48:49.063886 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:49.063890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:49.063932 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:49.063993 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:49.064262 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.064762 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.065758 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.066137 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.066236 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:49.066281 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:49.066442 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 13:48:49.066507 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 13:48:49.066615 32656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:49.063168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:42809 (local address 127.12.170.67:39831)
0926 13:48:49.063618 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:49.063628 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:49.063781 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:49.064011 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:49.064014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:49.064029 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:49.064152 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:49.064161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.064883 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.064886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:49.065641 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:49.065649 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.066238 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.066241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:49.066331 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:49.066353 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 13:48:49.066413 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 13:48:49.066473 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":118,"spinlock_wait_cycles":768,"thread_start_us":151,"threads_started":1}
I20260926 13:48:49.067098 32316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ee5cd2088af4a5abd9473873ee95f2c" candidate_uuid: "8163c130e4c843cab2076141593eae56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b573d7b06bff4419b56744f4599681db" is_pre_election: true
W20260926 13:48:49.067191 32631 tablet.cc:2369] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:48:49.067251 32450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ee5cd2088af4a5abd9473873ee95f2c" candidate_uuid: "8163c130e4c843cab2076141593eae56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a91f7c8f8a04e18ab7fcb4a46a49787" is_pre_election: true
I20260926 13:48:49.067414 32316 raft_consensus.cc:2454] T 7ee5cd2088af4a5abd9473873ee95f2c P b573d7b06bff4419b56744f4599681db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8163c130e4c843cab2076141593eae56 in term 0.
I20260926 13:48:49.067518 32450 raft_consensus.cc:2454] T 7ee5cd2088af4a5abd9473873ee95f2c P 4a91f7c8f8a04e18ab7fcb4a46a49787 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8163c130e4c843cab2076141593eae56 in term 0.
I20260926 13:48:49.067884 32518 leader_election.cc:302] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8163c130e4c843cab2076141593eae56, b573d7b06bff4419b56744f4599681db; no voters:
I20260926 13:48:49.068099 32654 raft_consensus.cc:2781] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:48:49.068176 32654 raft_consensus.cc:482] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:48:49.068225 32654 raft_consensus.cc:3037] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:49.069196 32654 raft_consensus.cc:504] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b573d7b06bff4419b56744f4599681db" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 46509 } } peers { permanent_uuid: "8163c130e4c843cab2076141593eae56" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42103 } } peers { permanent_uuid: "4a91f7c8f8a04e18ab7fcb4a46a49787" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42809 } }
I20260926 13:48:49.069518 32654 leader_election.cc:288] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56 [CANDIDATE]: Term 1 election: Requested vote from peers b573d7b06bff4419b56744f4599681db (127.12.170.65:46509), 4a91f7c8f8a04e18ab7fcb4a46a49787 (127.12.170.66:42809)
I20260926 13:48:49.069839 32316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ee5cd2088af4a5abd9473873ee95f2c" candidate_uuid: "8163c130e4c843cab2076141593eae56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b573d7b06bff4419b56744f4599681db"
I20260926 13:48:49.069835 32450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ee5cd2088af4a5abd9473873ee95f2c" candidate_uuid: "8163c130e4c843cab2076141593eae56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a91f7c8f8a04e18ab7fcb4a46a49787"
I20260926 13:48:49.069940 32450 raft_consensus.cc:3037] T 7ee5cd2088af4a5abd9473873ee95f2c P 4a91f7c8f8a04e18ab7fcb4a46a49787 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:49.069939 32316 raft_consensus.cc:3037] T 7ee5cd2088af4a5abd9473873ee95f2c P b573d7b06bff4419b56744f4599681db [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:49.070863 32316 raft_consensus.cc:2454] T 7ee5cd2088af4a5abd9473873ee95f2c P b573d7b06bff4419b56744f4599681db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8163c130e4c843cab2076141593eae56 in term 1.
I20260926 13:48:49.070942 32450 raft_consensus.cc:2454] T 7ee5cd2088af4a5abd9473873ee95f2c P 4a91f7c8f8a04e18ab7fcb4a46a49787 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8163c130e4c843cab2076141593eae56 in term 1.
I20260926 13:48:49.071128 32518 leader_election.cc:302] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8163c130e4c843cab2076141593eae56, b573d7b06bff4419b56744f4599681db; no voters:
I20260926 13:48:49.071295 32654 raft_consensus.cc:2781] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:48:49.071499 32654 raft_consensus.cc:686] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56 [term 1 LEADER]: Becoming Leader. State: Replica: 8163c130e4c843cab2076141593eae56, State: Running, Role: LEADER
I20260926 13:48:49.071827 32654 consensus_queue.cc:237] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b573d7b06bff4419b56744f4599681db" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 46509 } } peers { permanent_uuid: "8163c130e4c843cab2076141593eae56" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42103 } } peers { permanent_uuid: "4a91f7c8f8a04e18ab7fcb4a46a49787" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42809 } }
I20260926 13:48:49.074653 32171 catalog_manager.cc:5495] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8163c130e4c843cab2076141593eae56 (127.12.170.67). New cstate: current_term: 1 leader_uuid: "8163c130e4c843cab2076141593eae56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b573d7b06bff4419b56744f4599681db" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 46509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8163c130e4c843cab2076141593eae56" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 42103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a91f7c8f8a04e18ab7fcb4a46a49787" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42809 } health_report { overall_health: UNKNOWN } } }
I20260926 13:48:49.095057 32644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:49.091809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44270 (local address 127.12.170.67:42103)
0926 13:48:49.091930 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:49.091933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:49.091949 (+ 16us) server_negotiation.cc:407] Connection header received
0926 13:48:49.092082 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:49.092085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:49.092146 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:49.092239 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:48:49.092446 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.092940 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.093716 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.093929 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.094008 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:49.094299 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:48:49.094718 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:48:49.094817 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:49.094899 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 13:48:49.094950 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 13:48:49.100730 32316 raft_consensus.cc:1261] T 7ee5cd2088af4a5abd9473873ee95f2c P b573d7b06bff4419b56744f4599681db [term 1 FOLLOWER]: Refusing update from remote peer 8163c130e4c843cab2076141593eae56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:48:49.100730 32450 raft_consensus.cc:1261] T 7ee5cd2088af4a5abd9473873ee95f2c P 4a91f7c8f8a04e18ab7fcb4a46a49787 [term 1 FOLLOWER]: Refusing update from remote peer 8163c130e4c843cab2076141593eae56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:48:49.101389 32657 consensus_queue.cc:1035] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b573d7b06bff4419b56744f4599681db" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 46509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:48:49.101548 32654 consensus_queue.cc:1035] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a91f7c8f8a04e18ab7fcb4a46a49787" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 42809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:48:49.111822 32661 mvcc.cc:205] Tried to move back new op lower bound from 7333603447189245952 to 7333603447080353792. Current Snapshot: MvccSnapshot[applied={T|T < 7333603447189245952 or (T in {7333603447189245952})}]
I20260926 13:48:49.116066 32170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:48:49.116238 32170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:48:49.118502 32170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b2a61da731e2480780a67f743984088c: Sending DeleteTablet for 3 replicas of tablet 7ee5cd2088af4a5abd9473873ee95f2c
I20260926 13:48:49.119148 32296 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ee5cd2088af4a5abd9473873ee95f2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:48:49 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:33274
I20260926 13:48:49.119171 32564 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ee5cd2088af4a5abd9473873ee95f2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:48:49 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:44266
I20260926 13:48:49.119630 32430 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ee5cd2088af4a5abd9473873ee95f2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:48:49 UTC) from {username='kudu', principal='kudu/127.12.170.126@KRBTEST.COM'} at 127.0.0.1:49286
I20260926 13:48:49.119982 32670 tablet_replica.cc:330] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56: stopping tablet replica
I20260926 13:48:49.120265 32670 raft_consensus.cc:2229] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:48:49.120419 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 32233
I20260926 13:48:49.120913 32670 raft_consensus.cc:2258] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:48:49.122717 32670 ts_tablet_manager.cc:1905] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:48:49.125527 32670 ts_tablet_manager.cc:1918] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:48:49.125648 32670 log.cc:1196] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/7ee5cd2088af4a5abd9473873ee95f2c
I20260926 13:48:49.125981 32670 ts_tablet_manager.cc:1939] T 7ee5cd2088af4a5abd9473873ee95f2c P 8163c130e4c843cab2076141593eae56: Deleting consensus metadata
W20260926 13:48:49.126235 32156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.170.65:46509 (error 108)
I20260926 13:48:49.126957 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 32367
I20260926 13:48:49.127575 32157 catalog_manager.cc:4841] TS 8163c130e4c843cab2076141593eae56 (127.12.170.67:42103): tablet 7ee5cd2088af4a5abd9473873ee95f2c (table test-table [id=0af095d1f7714a5299d1292d5d2af9b9]) successfully deleted
I20260926 13:48:49.136472 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 32501
I20260926 13:48:49.137501 32645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:49.136650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:46509 (local address 127.0.0.1:33276)
0926 13:48:49.137344 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:49.137423 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.65:46509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":612}
W20260926 13:48:49.138034 32156 catalog_manager.cc:4584] TS b573d7b06bff4419b56744f4599681db (127.12.170.65:46509): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7ee5cd2088af4a5abd9473873ee95f2c: Network error: Client connection negotiation failed: client connection to 127.12.170.65:46509: connect: Connection refused (error 111)
I20260926 13:48:49.138518 32645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:49.138318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:42809 (local address 127.0.0.1:49298)
0926 13:48:49.138425 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:49.138469 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.66:42809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 13:48:49.141667 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 32139
2026-09-26T13:48:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2243 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T13:48:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:48:49Z Disabled control of system clock
I20260926 13:48:49.168679 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:37545
--webserver_interface=127.12.170.126
--webserver_port=0
--builtin_ntp_servers=127.12.170.84:46685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:37545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:48:49.271557 32676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:49.271831 32676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:49.271874 32676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:49.275694 32676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:48:49.275743 32676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:49.275756 32676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:49.275770 32676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:48:49.275784 32676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:48:49.280303 32676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:46685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:37545
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:37545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:49.281207 32676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:49.282428 32676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:49.290238 32681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:49.290362 32682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:49.290292 32676 server_base.cc:1034] running on GCE node
W20260926 13:48:49.290249 32684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:49.290978 32676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:49.292012 32676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:49.293192 32676 hybrid_clock.cc:648] HybridClock initialized: now 1790430529293177 us; error 26 us; skew 500 ppm
I20260926 13:48:49.294916 32676 webserver.cc:460] Webserver started at http://127.12.170.126:36481/ using document root <none> and password file <none>
I20260926 13:48:49.295506 32676 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:49.295578 32676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:49.295804 32676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:49.297672 32676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "81b8123778cf47508afdfd1aaeddda87"
format_stamp: "Formatted at 2026-09-26 13:48:49 on dist-test-slave-hc1k"
server_key: "c49936ebd502a5f3cdc1d496007e4eec"
server_key_iv: "ac6482d3350924a67085932bf40ec561"
server_key_version: "encryptionkey@0"
I20260926 13:48:49.298197 32676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "81b8123778cf47508afdfd1aaeddda87"
format_stamp: "Formatted at 2026-09-26 13:48:49 on dist-test-slave-hc1k"
server_key: "c49936ebd502a5f3cdc1d496007e4eec"
server_key_iv: "ac6482d3350924a67085932bf40ec561"
server_key_version: "encryptionkey@0"
I20260926 13:48:49.301582 32676 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:48:49.303865 32690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:49.305007 32676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:48:49.305125 32676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "81b8123778cf47508afdfd1aaeddda87"
format_stamp: "Formatted at 2026-09-26 13:48:49 on dist-test-slave-hc1k"
server_key: "c49936ebd502a5f3cdc1d496007e4eec"
server_key_iv: "ac6482d3350924a67085932bf40ec561"
server_key_version: "encryptionkey@0"
I20260926 13:48:49.305217 32676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:49.325205 32676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:49.325845 32676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:49.326009 32676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:49.332397 32676 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:37545
I20260926 13:48:49.332427 32742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:37545 every 8 connection(s)
I20260926 13:48:49.333258 32676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:48:49.336192 32743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:49.341341 32743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81b8123778cf47508afdfd1aaeddda87: Bootstrap starting.
I20260926 13:48:49.343358 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 32676
I20260926 13:48:49.343469 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:48:49.343683 32743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81b8123778cf47508afdfd1aaeddda87: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:49.343751 12969 external_mini_cluster.cc:1428] Setting key eeb31cc1ff288fd9e7ebfebc2a5464c6
I20260926 13:48:49.344420 32743 log.cc:824] T 00000000000000000000000000000000 P 81b8123778cf47508afdfd1aaeddda87: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:49.346216 32743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81b8123778cf47508afdfd1aaeddda87: No bootstrap required, opened a new log
I20260926 13:48:49.349884 32743 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81b8123778cf47508afdfd1aaeddda87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b8123778cf47508afdfd1aaeddda87" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 37545 } }
I20260926 13:48:49.350096 32743 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81b8123778cf47508afdfd1aaeddda87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:49.350149 32743 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81b8123778cf47508afdfd1aaeddda87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b8123778cf47508afdfd1aaeddda87, State: Initialized, Role: FOLLOWER
I20260926 13:48:49.350549 32743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81b8123778cf47508afdfd1aaeddda87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b8123778cf47508afdfd1aaeddda87" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 37545 } }
I20260926 13:48:49.350679 32743 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81b8123778cf47508afdfd1aaeddda87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:48:49.350739 32743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81b8123778cf47508afdfd1aaeddda87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:48:49.350839 32743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81b8123778cf47508afdfd1aaeddda87 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:49.351634 32743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81b8123778cf47508afdfd1aaeddda87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b8123778cf47508afdfd1aaeddda87" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 37545 } }
I20260926 13:48:49.351974 32743 leader_election.cc:302] T 00000000000000000000000000000000 P 81b8123778cf47508afdfd1aaeddda87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81b8123778cf47508afdfd1aaeddda87; no voters:
I20260926 13:48:49.352234 32743 leader_election.cc:288] T 00000000000000000000000000000000 P 81b8123778cf47508afdfd1aaeddda87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:48:49.352351 32748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81b8123778cf47508afdfd1aaeddda87 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:48:49.352449 32747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:49.344835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45654 (local address 127.12.170.126:37545)
0926 13:48:49.345443 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:49.345458 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:49.345491 (+ 33us) server_negotiation.cc:407] Connection header received
0926 13:48:49.346083 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:49.346098 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:49.346435 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:49.346740 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:49.347124 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.348434 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.349266 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.349749 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.349892 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:49.350614 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:49.350620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:49.350633 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:48:49.351070 (+ 437us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:49.351122 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:48:49.351141 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:49.351222 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:49.351364 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 13:48:49.351607 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":104,"threads_started":1}
I20260926 13:48:49.352555 32748 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81b8123778cf47508afdfd1aaeddda87 [term 1 LEADER]: Becoming Leader. State: Replica: 81b8123778cf47508afdfd1aaeddda87, State: Running, Role: LEADER
I20260926 13:48:49.352859 32748 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81b8123778cf47508afdfd1aaeddda87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b8123778cf47508afdfd1aaeddda87" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 37545 } }
I20260926 13:48:49.353366 32743 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81b8123778cf47508afdfd1aaeddda87 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:48:49.353960 32749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81b8123778cf47508afdfd1aaeddda87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81b8123778cf47508afdfd1aaeddda87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b8123778cf47508afdfd1aaeddda87" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 37545 } } }
I20260926 13:48:49.354063 32749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81b8123778cf47508afdfd1aaeddda87 [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:49.354286 32750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81b8123778cf47508afdfd1aaeddda87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81b8123778cf47508afdfd1aaeddda87. Latest consensus state: current_term: 1 leader_uuid: "81b8123778cf47508afdfd1aaeddda87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81b8123778cf47508afdfd1aaeddda87" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 37545 } } }
I20260926 13:48:49.354372 32750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81b8123778cf47508afdfd1aaeddda87 [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:49.354748 32754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:48:49.357679 32754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:48:49.363420 32754 catalog_manager.cc:1347] Generated new cluster ID: 9a945ba697384080859cc65683addf36
I20260926 13:48:49.363548 32754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:48:49.371138 32754 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:48:49.372068 32754 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:48:49.382668 32754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81b8123778cf47508afdfd1aaeddda87: Generated new TSK 0
I20260926 13:48:49.383342 32754 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:48:49.394878 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.65:0
--local_ip_for_outbound_sockets=127.12.170.65
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:37545
--builtin_ntp_servers=127.12.170.84:46685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:48:49.505522 32767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:49.505774 32767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:49.505811 32767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:49.509939 32767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:49.510010 32767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:49.510074 32767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.65
I20260926 13:48:49.514941 32767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:46685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.170.65
--webserver_port=0
--tserver_master_addrs=127.12.170.126:37545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.65
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:49.516012 32767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:49.517376 32767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:49.526765 305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:49.526801 307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:49.526880 32767 server_base.cc:1034] running on GCE node
W20260926 13:48:49.526786 304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:49.527416 32767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:49.528031 32767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:49.529181 32767 hybrid_clock.cc:648] HybridClock initialized: now 1790430529529162 us; error 29 us; skew 500 ppm
I20260926 13:48:49.530900 32767 webserver.cc:460] Webserver started at http://127.12.170.65:45589/ using document root <none> and password file <none>
I20260926 13:48:49.531436 32767 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:49.531497 32767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:49.531718 32767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:49.533697 32767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-0/data/instance:
uuid: "c311ccc777a64e71adc1dacab77a1059"
format_stamp: "Formatted at 2026-09-26 13:48:49 on dist-test-slave-hc1k"
server_key: "cc2d56f99bbb2498a0be75fade61a6f2"
server_key_iv: "ef2d87c3b97688597d66e4d642be4a16"
server_key_version: "encryptionkey@0"
I20260926 13:48:49.534214 32767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance:
uuid: "c311ccc777a64e71adc1dacab77a1059"
format_stamp: "Formatted at 2026-09-26 13:48:49 on dist-test-slave-hc1k"
server_key: "cc2d56f99bbb2498a0be75fade61a6f2"
server_key_iv: "ef2d87c3b97688597d66e4d642be4a16"
server_key_version: "encryptionkey@0"
I20260926 13:48:49.537678 32767 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 13:48:49.540069 313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:49.541281 32767 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:48:49.541388 32767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-0/wal
uuid: "c311ccc777a64e71adc1dacab77a1059"
format_stamp: "Formatted at 2026-09-26 13:48:49 on dist-test-slave-hc1k"
server_key: "cc2d56f99bbb2498a0be75fade61a6f2"
server_key_iv: "ef2d87c3b97688597d66e4d642be4a16"
server_key_version: "encryptionkey@0"
I20260926 13:48:49.541486 32767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:49.556231 32767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:49.556807 32767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:49.556942 32767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:49.557425 32767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:48:49.558269 32767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:48:49.558320 32767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:49.558358 32767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:48:49.558378 32767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:49.566256 32767 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.65:40847
I20260926 13:48:49.566332 426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.65:40847 every 8 connection(s)
I20260926 13:48:49.567080 32767 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-0/data/info.pb
I20260926 13:48:49.573658 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 32767
I20260926 13:48:49.573781 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-0/wal/instance
I20260926 13:48:49.574043 12969 external_mini_cluster.cc:1428] Setting key e6077cd3b1910eb28a945fd0f44b8cd8
I20260926 13:48:49.575762 32747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:49.568739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.65:35839 (local address 127.12.170.126:37545)
0926 13:48:49.568881 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:49.568884 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:49.569852 (+ 968us) server_negotiation.cc:407] Connection header received
0926 13:48:49.571221 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:49.571224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:49.571284 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:49.571365 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:49.572251 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.572780 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.573868 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.574093 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.574183 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:49.575299 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:49.575304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:49.575306 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:48:49.575319 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:49.575342 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:48:49.575352 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:49.575419 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:49.575567 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 13:48:49.575619 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 13:48:49.576085 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.66:0
--local_ip_for_outbound_sockets=127.12.170.66
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:37545
--builtin_ntp_servers=127.12.170.84:46685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 13:48:49.576855 429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:49.569111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:37545 (local address 127.12.170.65:35839)
0926 13:48:49.569708 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:49.569746 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:49.571028 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:49.571530 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:49.571537 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:49.571617 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:48:49.572117 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:49.572129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.572892 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.572894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:49.573750 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:49.573759 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.574234 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.574239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:49.574457 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:49.575034 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:48:49.575036 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:48:49.575073 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:48:49.575131 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:48:49.575432 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:48:49.575440 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:48:49.575446 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 13:48:49.575571 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 13:48:49.575928 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":119,"threads_started":1}
I20260926 13:48:49.577955 427 heartbeater.cc:347] Connected to a master server at 127.12.170.126:37545
I20260926 13:48:49.578153 427 heartbeater.cc:464] Registering TS with master...
I20260926 13:48:49.578605 427 heartbeater.cc:511] Master 127.12.170.126:37545 requested a full tablet report, sending...
I20260926 13:48:49.579977 32707 ts_manager.cc:194] Registered new tserver with Master: c311ccc777a64e71adc1dacab77a1059 (127.12.170.65:40847)
I20260926 13:48:49.581086 32707 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.170.65:35839
W20260926 13:48:49.681203 430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:49.681440 430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:49.681486 430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:49.685431 430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:49.685477 430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:49.685545 430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.66
I20260926 13:48:49.690060 430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:46685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.170.66
--webserver_port=0
--tserver_master_addrs=127.12.170.126:37545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.66
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:49.690981 430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:49.692298 430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:49.701076 438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:49.701095 435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:49.701157 436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:49.701304 430 server_base.cc:1034] running on GCE node
I20260926 13:48:49.701673 430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:49.702224 430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:49.703400 430 hybrid_clock.cc:648] HybridClock initialized: now 1790430529703379 us; error 30 us; skew 500 ppm
I20260926 13:48:49.705159 430 webserver.cc:460] Webserver started at http://127.12.170.66:37389/ using document root <none> and password file <none>
I20260926 13:48:49.705715 430 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:49.705785 430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:49.705981 430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:49.708003 430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-1/data/instance:
uuid: "c651376f6a9e4e50b9922d3c3df4dc1e"
format_stamp: "Formatted at 2026-09-26 13:48:49 on dist-test-slave-hc1k"
server_key: "6a142b11e35d5db151ee81f1774a2f41"
server_key_iv: "9c30b91da532ef1c0168612e43ea0d48"
server_key_version: "encryptionkey@0"
I20260926 13:48:49.708596 430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance:
uuid: "c651376f6a9e4e50b9922d3c3df4dc1e"
format_stamp: "Formatted at 2026-09-26 13:48:49 on dist-test-slave-hc1k"
server_key: "6a142b11e35d5db151ee81f1774a2f41"
server_key_iv: "9c30b91da532ef1c0168612e43ea0d48"
server_key_version: "encryptionkey@0"
I20260926 13:48:49.712107 430 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:48:49.714376 444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:49.715553 430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:48:49.715692 430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-1/wal
uuid: "c651376f6a9e4e50b9922d3c3df4dc1e"
format_stamp: "Formatted at 2026-09-26 13:48:49 on dist-test-slave-hc1k"
server_key: "6a142b11e35d5db151ee81f1774a2f41"
server_key_iv: "9c30b91da532ef1c0168612e43ea0d48"
server_key_version: "encryptionkey@0"
I20260926 13:48:49.715792 430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:49.745486 430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:49.746165 430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:49.746340 430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:49.746909 430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:48:49.747846 430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:48:49.747895 430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:49.747949 430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:48:49.747975 430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:49.756175 430 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.66:46359
I20260926 13:48:49.756208 557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.66:46359 every 8 connection(s)
I20260926 13:48:49.756996 430 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-1/data/info.pb
I20260926 13:48:49.762814 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 430
I20260926 13:48:49.762933 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-1/wal/instance
I20260926 13:48:49.763235 12969 external_mini_cluster.cc:1428] Setting key 403e013bc977779b7bc4abdb5d60056b
I20260926 13:48:49.765192 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.170.67:0
--local_ip_for_outbound_sockets=127.12.170.67
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:37545
--builtin_ntp_servers=127.12.170.84:46685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 13:48:49.766044 32747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:49.758650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:34931 (local address 127.12.170.126:37545)
0926 13:48:49.758792 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:49.758797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:49.759823 (+ 1026us) server_negotiation.cc:407] Connection header received
0926 13:48:49.761095 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:49.761098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:49.761144 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:49.761216 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:49.762006 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.762520 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.763982 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.764244 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.764356 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:49.765555 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:49.765559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:49.765562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:48:49.765577 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:49.765599 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:48:49.765608 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:49.765671 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:49.765829 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 13:48:49.765891 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 13:48:49.766837 560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:49.759057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:37545 (local address 127.12.170.66:34931)
0926 13:48:49.759683 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:49.759728 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:49.760920 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:49.761360 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:49.761366 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:49.761446 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:48:49.761902 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:49.761915 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.762629 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.762633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:49.763874 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:49.763883 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.764397 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.764402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:49.764604 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:49.765255 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:48:49.765258 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:48:49.765315 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:48:49.765398 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:48:49.765681 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:48:49.765690 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:48:49.765696 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 13:48:49.765819 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 13:48:49.766078 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":118,"threads_started":1}
I20260926 13:48:49.767913 558 heartbeater.cc:347] Connected to a master server at 127.12.170.126:37545
I20260926 13:48:49.768126 558 heartbeater.cc:464] Registering TS with master...
I20260926 13:48:49.768664 558 heartbeater.cc:511] Master 127.12.170.126:37545 requested a full tablet report, sending...
I20260926 13:48:49.769726 32707 ts_manager.cc:194] Registered new tserver with Master: c651376f6a9e4e50b9922d3c3df4dc1e (127.12.170.66:46359)
I20260926 13:48:49.770448 32707 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.170.66:34931
W20260926 13:48:49.870682 561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:49.870909 561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:49.870944 561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:49.874830 561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:49.874879 561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:49.874930 561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.170.67
I20260926 13:48:49.879442 561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:46685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.170.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.170.67
--webserver_port=0
--tserver_master_addrs=127.12.170.126:37545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.170.67
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:49.880440 561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:49.881793 561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:49.891192 566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:49.891260 561 server_base.cc:1034] running on GCE node
W20260926 13:48:49.891299 569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:49.891278 567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:49.891836 561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:49.892469 561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:49.893635 561 hybrid_clock.cc:648] HybridClock initialized: now 1790430529893615 us; error 32 us; skew 500 ppm
I20260926 13:48:49.895392 561 webserver.cc:460] Webserver started at http://127.12.170.67:44171/ using document root <none> and password file <none>
I20260926 13:48:49.895979 561 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:49.896044 561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:49.896236 561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:49.898185 561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-2/data/instance:
uuid: "2fe107e57c2e445f931943e7df5ba000"
format_stamp: "Formatted at 2026-09-26 13:48:49 on dist-test-slave-hc1k"
server_key: "5d0278929fd2a1d5e8a2b2a592985bc7"
server_key_iv: "c2e978f963ffe06eafe6ec55fc3c3bb7"
server_key_version: "encryptionkey@0"
I20260926 13:48:49.898713 561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance:
uuid: "2fe107e57c2e445f931943e7df5ba000"
format_stamp: "Formatted at 2026-09-26 13:48:49 on dist-test-slave-hc1k"
server_key: "5d0278929fd2a1d5e8a2b2a592985bc7"
server_key_iv: "c2e978f963ffe06eafe6ec55fc3c3bb7"
server_key_version: "encryptionkey@0"
I20260926 13:48:49.902323 561 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 13:48:49.904695 575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:49.905872 561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:48:49.905975 561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-2/wal
uuid: "2fe107e57c2e445f931943e7df5ba000"
format_stamp: "Formatted at 2026-09-26 13:48:49 on dist-test-slave-hc1k"
server_key: "5d0278929fd2a1d5e8a2b2a592985bc7"
server_key_iv: "c2e978f963ffe06eafe6ec55fc3c3bb7"
server_key_version: "encryptionkey@0"
I20260926 13:48:49.906064 561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:49.919916 561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:49.920569 561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:49.920732 561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:49.921264 561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:48:49.922140 561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:48:49.922189 561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:49.922240 561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:48:49.922264 561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:49.931423 561 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.67:43807
I20260926 13:48:49.931473 688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.67:43807 every 8 connection(s)
I20260926 13:48:49.932391 561 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-2/data/info.pb
I20260926 13:48:49.941071 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 561
I20260926 13:48:49.941184 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-2/wal/instance
I20260926 13:48:49.941336 32747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:49.934162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.67:57257 (local address 127.12.170.126:37545)
0926 13:48:49.934310 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:49.934316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:49.935450 (+ 1134us) server_negotiation.cc:407] Connection header received
0926 13:48:49.936859 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:49.936862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:49.936911 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:49.936984 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:49.937849 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.938383 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.939535 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.939788 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.939867 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:49.940885 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:49.940888 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:49.940891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:48:49.940905 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:49.940928 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:48:49.940938 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:49.941015 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:49.941128 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 13:48:49.941192 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 13:48:49.941546 12969 external_mini_cluster.cc:1428] Setting key 772852b8b5f88bffc288988fb8b271ed
I20260926 13:48:49.942265 691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:49.934621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:37545 (local address 127.12.170.67:57257)
0926 13:48:49.935304 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:49.935346 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:49.936636 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:49.937135 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:49.937142 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:49.937226 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:48:49.937704 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:49.937717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.938527 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.938530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:49.939424 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:49.939434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.939912 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.939915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:49.940052 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:49.940605 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:48:49.940607 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:48:49.940647 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:48:49.940710 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:48:49.941014 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:48:49.941019 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:48:49.941023 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:48:49.941110 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 13:48:49.941376 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":108,"threads_started":1}
I20260926 13:48:49.943348 689 heartbeater.cc:347] Connected to a master server at 127.12.170.126:37545
I20260926 13:48:49.943590 689 heartbeater.cc:464] Registering TS with master...
I20260926 13:48:49.944129 689 heartbeater.cc:511] Master 127.12.170.126:37545 requested a full tablet report, sending...
I20260926 13:48:49.945118 32706 ts_manager.cc:194] Registered new tserver with Master: 2fe107e57c2e445f931943e7df5ba000 (127.12.170.67:43807)
I20260926 13:48:49.945706 32706 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.170.67:57257
I20260926 13:48:49.955428 12969 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:48:49.960812 32747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:49.957270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45668 (local address 127.12.170.126:37545)
0926 13:48:49.957387 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:49.957392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:49.957582 (+ 190us) server_negotiation.cc:407] Connection header received
0926 13:48:49.957694 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:49.957697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:49.957743 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:49.957805 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:49.958080 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.958598 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.959404 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.959625 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.959725 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:49.960440 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:49.960444 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:49.960447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:48:49.960459 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:49.960480 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:48:49.960488 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:49.960543 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:49.960624 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 13:48:49.960677 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 13:48:49.963574 32706 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:48:49.966528 32706 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:48:49.979902 704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:49.975177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41968 (local address 127.12.170.66:46359)
0926 13:48:49.975908 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:49.975921 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:49.975944 (+ 23us) server_negotiation.cc:407] Connection header received
0926 13:48:49.976015 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:49.976020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:49.976182 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:49.976305 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:49.976528 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.977245 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.978534 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.979178 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.979281 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:49.979462 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:49.979612 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 13:48:49.979737 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":92,"threads_started":1}
I20260926 13:48:49.979897 702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:49.974686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:46359 (local address 127.0.0.1:41968)
0926 13:48:49.975119 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:49.975131 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:49.975206 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:49.976302 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:49.976304 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:49.976320 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:49.976419 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:49.976426 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.977356 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.977359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:49.978411 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:49.978419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.979290 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.979294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:49.979394 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:49.979471 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 13:48:49.979572 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 13:48:49.979745 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":27,"thread_start_us":100,"threads_started":1}
I20260926 13:48:49.980171 700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:49.974414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:43807 (local address 127.0.0.1:48604)
0926 13:48:49.974728 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:49.974780 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:49.974908 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:49.975546 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:49.975552 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:49.975576 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:49.975757 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:49.975765 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.977247 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.977250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:49.978453 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:49.978463 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.979419 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.979422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:49.979520 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:49.979539 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 13:48:49.979579 (+ 40us) client_negotiation.cc:240] Negotiation successful
0926 13:48:49.980065 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":424,"thread_start_us":64,"threads_started":1}
I20260926 13:48:49.980177 701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:49.974599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48604 (local address 127.12.170.67:43807)
0926 13:48:49.975038 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:49.975050 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:49.975074 (+ 24us) server_negotiation.cc:407] Connection header received
0926 13:48:49.975150 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:49.975155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:49.975372 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:49.975913 (+ 541us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:49.976041 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.977136 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.978584 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.979287 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.979720 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:49.979888 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:49.979937 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 13:48:49.980025 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":153,"threads_started":1}
I20260926 13:48:49.980553 703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:49.975098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:40847 (local address 127.0.0.1:51128)
0926 13:48:49.975562 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:49.975574 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:49.975646 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:49.976373 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:49.976376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:49.976393 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:49.976476 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:49.976481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.977878 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.977882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:49.978951 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:49.978960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.980199 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.980202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:49.980296 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:49.980319 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 13:48:49.980377 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 13:48:49.980438 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":110,"threads_started":1}
I20260926 13:48:49.981177 705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:49.975510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51128 (local address 127.12.170.65:40847)
0926 13:48:49.975912 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:49.975923 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:49.975944 (+ 21us) server_negotiation.cc:407] Connection header received
0926 13:48:49.976016 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:49.976021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:49.976221 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:49.976601 (+ 380us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:49.976724 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.977764 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.979068 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:49.979805 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:49.979899 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:49.980920 (+ 1021us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:49.980967 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 13:48:49.981051 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":90,"threads_started":1}
I20260926 13:48:49.983067 492 tablet_service.cc:1467] Processing CreateTablet for tablet 01559c492e19493ca9caf2e9288272a1 (DEFAULT_TABLE table=test-table [id=11623fce8eb34552bcdfa41a7141a9b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:48:49.983069 623 tablet_service.cc:1467] Processing CreateTablet for tablet 01559c492e19493ca9caf2e9288272a1 (DEFAULT_TABLE table=test-table [id=11623fce8eb34552bcdfa41a7141a9b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:48:49.983479 361 tablet_service.cc:1467] Processing CreateTablet for tablet 01559c492e19493ca9caf2e9288272a1 (DEFAULT_TABLE table=test-table [id=11623fce8eb34552bcdfa41a7141a9b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:48:49.984171 492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01559c492e19493ca9caf2e9288272a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:49.984171 623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01559c492e19493ca9caf2e9288272a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:49.984338 361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01559c492e19493ca9caf2e9288272a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:49.990640 706 tablet_bootstrap.cc:492] T 01559c492e19493ca9caf2e9288272a1 P c311ccc777a64e71adc1dacab77a1059: Bootstrap starting.
I20260926 13:48:49.991335 707 tablet_bootstrap.cc:492] T 01559c492e19493ca9caf2e9288272a1 P 2fe107e57c2e445f931943e7df5ba000: Bootstrap starting.
I20260926 13:48:49.991942 708 tablet_bootstrap.cc:492] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e: Bootstrap starting.
I20260926 13:48:49.993029 706 tablet_bootstrap.cc:654] T 01559c492e19493ca9caf2e9288272a1 P c311ccc777a64e71adc1dacab77a1059: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:49.993459 707 tablet_bootstrap.cc:654] T 01559c492e19493ca9caf2e9288272a1 P 2fe107e57c2e445f931943e7df5ba000: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:49.993562 708 tablet_bootstrap.cc:654] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:49.993917 706 log.cc:824] T 01559c492e19493ca9caf2e9288272a1 P c311ccc777a64e71adc1dacab77a1059: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:49.994200 707 log.cc:824] T 01559c492e19493ca9caf2e9288272a1 P 2fe107e57c2e445f931943e7df5ba000: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:49.994261 708 log.cc:824] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:49.995987 706 tablet_bootstrap.cc:492] T 01559c492e19493ca9caf2e9288272a1 P c311ccc777a64e71adc1dacab77a1059: No bootstrap required, opened a new log
I20260926 13:48:49.995987 707 tablet_bootstrap.cc:492] T 01559c492e19493ca9caf2e9288272a1 P 2fe107e57c2e445f931943e7df5ba000: No bootstrap required, opened a new log
I20260926 13:48:49.996178 707 ts_tablet_manager.cc:1397] T 01559c492e19493ca9caf2e9288272a1 P 2fe107e57c2e445f931943e7df5ba000: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 13:48:49.996178 706 ts_tablet_manager.cc:1397] T 01559c492e19493ca9caf2e9288272a1 P c311ccc777a64e71adc1dacab77a1059: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 13:48:49.996205 708 tablet_bootstrap.cc:492] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e: No bootstrap required, opened a new log
I20260926 13:48:49.996389 708 ts_tablet_manager.cc:1397] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 13:48:49.999454 708 raft_consensus.cc:348] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe107e57c2e445f931943e7df5ba000" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 43807 } } peers { permanent_uuid: "c311ccc777a64e71adc1dacab77a1059" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40847 } } peers { permanent_uuid: "c651376f6a9e4e50b9922d3c3df4dc1e" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 46359 } }
I20260926 13:48:49.999711 708 raft_consensus.cc:374] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:49.999789 708 raft_consensus.cc:729] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c651376f6a9e4e50b9922d3c3df4dc1e, State: Initialized, Role: FOLLOWER
I20260926 13:48:50.000241 708 consensus_queue.cc:260] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe107e57c2e445f931943e7df5ba000" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 43807 } } peers { permanent_uuid: "c311ccc777a64e71adc1dacab77a1059" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40847 } } peers { permanent_uuid: "c651376f6a9e4e50b9922d3c3df4dc1e" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 46359 } }
I20260926 13:48:50.000373 707 raft_consensus.cc:348] T 01559c492e19493ca9caf2e9288272a1 P 2fe107e57c2e445f931943e7df5ba000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe107e57c2e445f931943e7df5ba000" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 43807 } } peers { permanent_uuid: "c311ccc777a64e71adc1dacab77a1059" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40847 } } peers { permanent_uuid: "c651376f6a9e4e50b9922d3c3df4dc1e" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 46359 } }
I20260926 13:48:50.000370 706 raft_consensus.cc:348] T 01559c492e19493ca9caf2e9288272a1 P c311ccc777a64e71adc1dacab77a1059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe107e57c2e445f931943e7df5ba000" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 43807 } } peers { permanent_uuid: "c311ccc777a64e71adc1dacab77a1059" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40847 } } peers { permanent_uuid: "c651376f6a9e4e50b9922d3c3df4dc1e" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 46359 } }
I20260926 13:48:50.000633 706 raft_consensus.cc:374] T 01559c492e19493ca9caf2e9288272a1 P c311ccc777a64e71adc1dacab77a1059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:50.000720 707 raft_consensus.cc:374] T 01559c492e19493ca9caf2e9288272a1 P 2fe107e57c2e445f931943e7df5ba000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:50.000722 706 raft_consensus.cc:729] T 01559c492e19493ca9caf2e9288272a1 P c311ccc777a64e71adc1dacab77a1059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c311ccc777a64e71adc1dacab77a1059, State: Initialized, Role: FOLLOWER
I20260926 13:48:50.000811 707 raft_consensus.cc:729] T 01559c492e19493ca9caf2e9288272a1 P 2fe107e57c2e445f931943e7df5ba000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fe107e57c2e445f931943e7df5ba000, State: Initialized, Role: FOLLOWER
I20260926 13:48:50.000926 558 heartbeater.cc:503] Master 127.12.170.126:37545 was elected leader, sending a full tablet report...
I20260926 13:48:50.001278 706 consensus_queue.cc:260] T 01559c492e19493ca9caf2e9288272a1 P c311ccc777a64e71adc1dacab77a1059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe107e57c2e445f931943e7df5ba000" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 43807 } } peers { permanent_uuid: "c311ccc777a64e71adc1dacab77a1059" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40847 } } peers { permanent_uuid: "c651376f6a9e4e50b9922d3c3df4dc1e" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 46359 } }
I20260926 13:48:50.001272 707 consensus_queue.cc:260] T 01559c492e19493ca9caf2e9288272a1 P 2fe107e57c2e445f931943e7df5ba000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe107e57c2e445f931943e7df5ba000" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 43807 } } peers { permanent_uuid: "c311ccc777a64e71adc1dacab77a1059" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40847 } } peers { permanent_uuid: "c651376f6a9e4e50b9922d3c3df4dc1e" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 46359 } }
I20260926 13:48:50.002287 706 ts_tablet_manager.cc:1428] T 01559c492e19493ca9caf2e9288272a1 P c311ccc777a64e71adc1dacab77a1059: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 13:48:50.002287 707 ts_tablet_manager.cc:1428] T 01559c492e19493ca9caf2e9288272a1 P 2fe107e57c2e445f931943e7df5ba000: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 13:48:50.002658 427 heartbeater.cc:503] Master 127.12.170.126:37545 was elected leader, sending a full tablet report...
I20260926 13:48:50.002827 689 heartbeater.cc:503] Master 127.12.170.126:37545 was elected leader, sending a full tablet report...
I20260926 13:48:50.003388 708 ts_tablet_manager.cc:1428] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e: Time spent starting tablet: real 0.007s user 0.001s sys 0.004s
W20260926 13:48:50.007822 559 tablet.cc:2369] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:48:50.068168 428 tablet.cc:2369] T 01559c492e19493ca9caf2e9288272a1 P c311ccc777a64e71adc1dacab77a1059: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:48:50.161046 712 raft_consensus.cc:482] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:48:50.161231 712 raft_consensus.cc:504] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe107e57c2e445f931943e7df5ba000" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 43807 } } peers { permanent_uuid: "c311ccc777a64e71adc1dacab77a1059" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40847 } } peers { permanent_uuid: "c651376f6a9e4e50b9922d3c3df4dc1e" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 46359 } }
I20260926 13:48:50.162320 712 leader_election.cc:288] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fe107e57c2e445f931943e7df5ba000 (127.12.170.67:43807), c311ccc777a64e71adc1dacab77a1059 (127.12.170.65:40847)
I20260926 13:48:50.165720 560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:50.162540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.67:43807 (local address 127.12.170.66:50253)
0926 13:48:50.162659 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:50.162673 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:50.162761 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:50.162978 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:50.162981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:50.162998 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:50.163110 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:50.163117 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:50.163863 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:50.163866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:50.164699 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:50.164708 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:50.165338 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:50.165341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:50.165442 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:50.165460 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 13:48:50.165522 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 13:48:50.165590 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":19}
I20260926 13:48:50.165720 701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:50.162643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:50253 (local address 127.12.170.67:43807)
0926 13:48:50.162766 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:50.162770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:50.162784 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:48:50.162840 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:50.162843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:50.162895 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:50.162954 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:50.163206 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:50.163762 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:50.164818 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:50.165231 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:50.165337 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:50.165386 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:50.165550 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 13:48:50.165619 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:48:50.166074 705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:50.162926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.170.66:35583 (local address 127.12.170.65:40847)
0926 13:48:50.163130 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:50.163134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:50.163230 (+ 96us) server_negotiation.cc:407] Connection header received
0926 13:48:50.163325 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:50.163328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:50.163371 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:50.163428 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:48:50.163700 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:50.164171 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:50.165177 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:50.165547 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:50.165645 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:50.165691 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:50.165891 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 13:48:50.165962 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260926 13:48:50.166388 715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:50.162793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:40847 (local address 127.12.170.66:35583)
0926 13:48:50.163138 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:50.163149 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:48:50.163238 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:48:50.163449 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:48:50.163452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:48:50.163466 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:48:50.163575 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:50.163580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:50.164267 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:50.164271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:50.165059 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:48:50.165068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:48:50.165650 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:48:50.165654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:48:50.165742 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:50.165761 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 13:48:50.166211 (+ 450us) client_negotiation.cc:240] Negotiation successful
0926 13:48:50.166281 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":19,"thread_start_us":128,"threads_started":1}
I20260926 13:48:50.166374 643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01559c492e19493ca9caf2e9288272a1" candidate_uuid: "c651376f6a9e4e50b9922d3c3df4dc1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fe107e57c2e445f931943e7df5ba000" is_pre_election: true
I20260926 13:48:50.166680 643 raft_consensus.cc:2454] T 01559c492e19493ca9caf2e9288272a1 P 2fe107e57c2e445f931943e7df5ba000 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c651376f6a9e4e50b9922d3c3df4dc1e in term 0.
I20260926 13:48:50.166975 381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01559c492e19493ca9caf2e9288272a1" candidate_uuid: "c651376f6a9e4e50b9922d3c3df4dc1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c311ccc777a64e71adc1dacab77a1059" is_pre_election: true
I20260926 13:48:50.167165 446 leader_election.cc:302] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fe107e57c2e445f931943e7df5ba000, c651376f6a9e4e50b9922d3c3df4dc1e; no voters:
I20260926 13:48:50.167227 381 raft_consensus.cc:2454] T 01559c492e19493ca9caf2e9288272a1 P c311ccc777a64e71adc1dacab77a1059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c651376f6a9e4e50b9922d3c3df4dc1e in term 0.
I20260926 13:48:50.167418 712 raft_consensus.cc:2781] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:48:50.167503 712 raft_consensus.cc:482] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:48:50.167546 712 raft_consensus.cc:3037] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:50.168520 712 raft_consensus.cc:504] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe107e57c2e445f931943e7df5ba000" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 43807 } } peers { permanent_uuid: "c311ccc777a64e71adc1dacab77a1059" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40847 } } peers { permanent_uuid: "c651376f6a9e4e50b9922d3c3df4dc1e" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 46359 } }
I20260926 13:48:50.168787 712 leader_election.cc:288] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e [CANDIDATE]: Term 1 election: Requested vote from peers 2fe107e57c2e445f931943e7df5ba000 (127.12.170.67:43807), c311ccc777a64e71adc1dacab77a1059 (127.12.170.65:40847)
I20260926 13:48:50.169092 643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01559c492e19493ca9caf2e9288272a1" candidate_uuid: "c651376f6a9e4e50b9922d3c3df4dc1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fe107e57c2e445f931943e7df5ba000"
I20260926 13:48:50.169129 381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01559c492e19493ca9caf2e9288272a1" candidate_uuid: "c651376f6a9e4e50b9922d3c3df4dc1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c311ccc777a64e71adc1dacab77a1059"
I20260926 13:48:50.169214 643 raft_consensus.cc:3037] T 01559c492e19493ca9caf2e9288272a1 P 2fe107e57c2e445f931943e7df5ba000 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:50.169224 381 raft_consensus.cc:3037] T 01559c492e19493ca9caf2e9288272a1 P c311ccc777a64e71adc1dacab77a1059 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:50.170413 381 raft_consensus.cc:2454] T 01559c492e19493ca9caf2e9288272a1 P c311ccc777a64e71adc1dacab77a1059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c651376f6a9e4e50b9922d3c3df4dc1e in term 1.
I20260926 13:48:50.170413 643 raft_consensus.cc:2454] T 01559c492e19493ca9caf2e9288272a1 P 2fe107e57c2e445f931943e7df5ba000 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c651376f6a9e4e50b9922d3c3df4dc1e in term 1.
I20260926 13:48:50.170689 446 leader_election.cc:302] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c311ccc777a64e71adc1dacab77a1059, c651376f6a9e4e50b9922d3c3df4dc1e; no voters:
I20260926 13:48:50.170828 712 raft_consensus.cc:2781] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:48:50.171039 712 raft_consensus.cc:686] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e [term 1 LEADER]: Becoming Leader. State: Replica: c651376f6a9e4e50b9922d3c3df4dc1e, State: Running, Role: LEADER
I20260926 13:48:50.171366 712 consensus_queue.cc:237] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe107e57c2e445f931943e7df5ba000" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 43807 } } peers { permanent_uuid: "c311ccc777a64e71adc1dacab77a1059" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40847 } } peers { permanent_uuid: "c651376f6a9e4e50b9922d3c3df4dc1e" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 46359 } }
I20260926 13:48:50.174338 32706 catalog_manager.cc:5495] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e reported cstate change: term changed from 0 to 1, leader changed from <none> to c651376f6a9e4e50b9922d3c3df4dc1e (127.12.170.66). New cstate: current_term: 1 leader_uuid: "c651376f6a9e4e50b9922d3c3df4dc1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe107e57c2e445f931943e7df5ba000" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 43807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c311ccc777a64e71adc1dacab77a1059" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c651376f6a9e4e50b9922d3c3df4dc1e" member_type: VOTER last_known_addr { host: "127.12.170.66" port: 46359 } health_report { overall_health: HEALTHY } } }
W20260926 13:48:50.183275 690 tablet.cc:2369] T 01559c492e19493ca9caf2e9288272a1 P 2fe107e57c2e445f931943e7df5ba000: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:48:50.214156 704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:50.210829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41974 (local address 127.12.170.66:46359)
0926 13:48:50.210966 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:50.210971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:50.210984 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:48:50.211058 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:50.211060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:50.211113 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:50.211204 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:48:50.211446 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:50.211987 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:50.212802 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:50.213023 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:50.213107 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:50.213375 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:48:50.213789 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:48:50.213919 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:50.213964 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 13:48:50.214014 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 13:48:50.219736 643 raft_consensus.cc:1261] T 01559c492e19493ca9caf2e9288272a1 P 2fe107e57c2e445f931943e7df5ba000 [term 1 FOLLOWER]: Refusing update from remote peer c651376f6a9e4e50b9922d3c3df4dc1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:48:50.219751 381 raft_consensus.cc:1261] T 01559c492e19493ca9caf2e9288272a1 P c311ccc777a64e71adc1dacab77a1059 [term 1 FOLLOWER]: Refusing update from remote peer c651376f6a9e4e50b9922d3c3df4dc1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:48:50.220446 716 consensus_queue.cc:1035] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fe107e57c2e445f931943e7df5ba000" member_type: VOTER last_known_addr { host: "127.12.170.67" port: 43807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:48:50.220619 712 consensus_queue.cc:1035] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c311ccc777a64e71adc1dacab77a1059" member_type: VOTER last_known_addr { host: "127.12.170.65" port: 40847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:48:50.226084 719 mvcc.cc:205] Tried to move back new op lower bound from 7333603451772903424 to 7333603451584323584. Current Snapshot: MvccSnapshot[applied={T|T < 7333603451772903424}]
I20260926 13:48:50.228518 721 mvcc.cc:205] Tried to move back new op lower bound from 7333603451772903424 to 7333603451584323584. Current Snapshot: MvccSnapshot[applied={T|T < 7333603451772903424}]
I20260926 13:48:50.228760 720 mvcc.cc:205] Tried to move back new op lower bound from 7333603451772903424 to 7333603451584323584. Current Snapshot: MvccSnapshot[applied={T|T < 7333603451772903424 or (T in {7333603451772903424})}]
I20260926 13:48:50.234989 32706 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:48:50.235126 32706 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:48:50.237257 32706 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81b8123778cf47508afdfd1aaeddda87: Sending DeleteTablet for 3 replicas of tablet 01559c492e19493ca9caf2e9288272a1
I20260926 13:48:50.237990 492 tablet_service.cc:1514] Processing DeleteTablet for tablet 01559c492e19493ca9caf2e9288272a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:48:50 UTC) from {username='slave'} at 127.0.0.1:41968
I20260926 13:48:50.238111 623 tablet_service.cc:1514] Processing DeleteTablet for tablet 01559c492e19493ca9caf2e9288272a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:48:50 UTC) from {username='slave'} at 127.0.0.1:48604
I20260926 13:48:50.238562 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 32767
I20260926 13:48:50.238571 729 tablet_replica.cc:330] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e: stopping tablet replica
I20260926 13:48:50.238689 361 tablet_service.cc:1514] Processing DeleteTablet for tablet 01559c492e19493ca9caf2e9288272a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:48:50 UTC) from {username='slave'} at 127.0.0.1:51128
I20260926 13:48:50.238845 729 raft_consensus.cc:2229] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:48:50.239233 729 raft_consensus.cc:2258] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:48:50.240933 730 tablet_replica.cc:330] T 01559c492e19493ca9caf2e9288272a1 P 2fe107e57c2e445f931943e7df5ba000: stopping tablet replica
I20260926 13:48:50.241221 730 raft_consensus.cc:2229] T 01559c492e19493ca9caf2e9288272a1 P 2fe107e57c2e445f931943e7df5ba000 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:48:50.241514 730 raft_consensus.cc:2258] T 01559c492e19493ca9caf2e9288272a1 P 2fe107e57c2e445f931943e7df5ba000 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:48:50.242708 729 ts_tablet_manager.cc:1905] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:48:50.243047 730 ts_tablet_manager.cc:1905] T 01559c492e19493ca9caf2e9288272a1 P 2fe107e57c2e445f931943e7df5ba000: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:48:50.245394 729 ts_tablet_manager.cc:1918] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:48:50.245549 729 log.cc:1196] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-1/wal/wals/01559c492e19493ca9caf2e9288272a1
I20260926 13:48:50.245733 730 ts_tablet_manager.cc:1918] T 01559c492e19493ca9caf2e9288272a1 P 2fe107e57c2e445f931943e7df5ba000: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:48:50.245906 730 log.cc:1196] T 01559c492e19493ca9caf2e9288272a1 P 2fe107e57c2e445f931943e7df5ba000: Deleting WAL directory at /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790430291939677-12969-0/minicluster-data/ts-2/wal/wals/01559c492e19493ca9caf2e9288272a1
I20260926 13:48:50.245908 729 ts_tablet_manager.cc:1939] T 01559c492e19493ca9caf2e9288272a1 P c651376f6a9e4e50b9922d3c3df4dc1e: Deleting consensus metadata
W20260926 13:48:50.246033 32692 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.170.65:40847 (error 108)
I20260926 13:48:50.246261 730 ts_tablet_manager.cc:1939] T 01559c492e19493ca9caf2e9288272a1 P 2fe107e57c2e445f931943e7df5ba000: Deleting consensus metadata
I20260926 13:48:50.246848 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 430
I20260926 13:48:50.247522 32692 catalog_manager.cc:4841] TS 2fe107e57c2e445f931943e7df5ba000 (127.12.170.67:43807): tablet 01559c492e19493ca9caf2e9288272a1 (table test-table [id=11623fce8eb34552bcdfa41a7141a9b5]) successfully deleted
I20260926 13:48:50.251773 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 561
I20260926 13:48:50.252769 703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:50.252443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.66:46359 (local address 127.0.0.1:41988)
0926 13:48:50.252614 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:50.252682 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.66:46359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260926 13:48:50.252949 32691 catalog_manager.cc:4584] TS c651376f6a9e4e50b9922d3c3df4dc1e (127.12.170.66:46359): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 01559c492e19493ca9caf2e9288272a1: Network error: Client connection negotiation failed: client connection to 127.12.170.66:46359: connect: Connection refused (error 111)
I20260926 13:48:50.253654 703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:50.253412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.65:40847 (local address 127.0.0.1:51144)
0926 13:48:50.253537 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 13:48:50.253593 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.65:40847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 13:48:50.256834 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 32676
2026-09-26T13:48:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1114 ms)
[----------] 6 tests from EncryptionPolicyTest (9986 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:48:50 dist-test-slave-hc1k krb5kdc[734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:48:50 dist-test-slave-hc1k krb5kdc[734](info): set up 2 sockets
Sep 26 13:48:50 dist-test-slave-hc1k krb5kdc[734](info): commencing operation
krb5kdc: starting...
W20260926 13:48:51.211486 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:48:51 dist-test-slave-hc1k krb5kdc[734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:48:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:48:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:48:51.324714 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43831
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:48:51.428778 747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:51.429025 747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:51.429080 747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:51.433018 747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:48:51.433073 747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:51.433099 747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:51.433120 747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:48:51.433149 747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:48:51.438139 747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43831
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:51.439056 747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:51.440023 747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:51.447877 753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:51.447902 752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:51.447877 755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:51.448153 747 server_base.cc:1034] running on GCE node
I20260926 13:48:51.448632 747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:51.449556 747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:51.450706 747 hybrid_clock.cc:648] HybridClock initialized: now 1790430531450704 us; error 36 us; skew 500 ppm
Sep 26 13:48:51 dist-test-slave-hc1k krb5kdc[734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:51.457372 747 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:48:51.458194 747 webserver.cc:460] Webserver started at http://127.0.0.1:45835/ using document root <none> and password file <none>
I20260926 13:48:51.458735 747 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:51.458801 747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:51.458978 747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:51.460938 747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "b9d233f37c0842d4a1a0db66d78fe7db"
format_stamp: "Formatted at 2026-09-26 13:48:51 on dist-test-slave-hc1k"
server_key: "361ca748122531c999b53ec7b4940875"
server_key_iv: "10235d5ed9f83d4c6ef1080fa733a7e3"
server_key_version: "encryptionkey@0"
I20260926 13:48:51.461447 747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "b9d233f37c0842d4a1a0db66d78fe7db"
format_stamp: "Formatted at 2026-09-26 13:48:51 on dist-test-slave-hc1k"
server_key: "361ca748122531c999b53ec7b4940875"
server_key_iv: "10235d5ed9f83d4c6ef1080fa733a7e3"
server_key_version: "encryptionkey@0"
I20260926 13:48:51.464941 747 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:48:51.467083 762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:51.468250 747 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:48:51.468354 747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "b9d233f37c0842d4a1a0db66d78fe7db"
format_stamp: "Formatted at 2026-09-26 13:48:51 on dist-test-slave-hc1k"
server_key: "361ca748122531c999b53ec7b4940875"
server_key_iv: "10235d5ed9f83d4c6ef1080fa733a7e3"
server_key_version: "encryptionkey@0"
I20260926 13:48:51.468442 747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:51.488852 747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:51.491504 747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:51.491683 747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:51.498104 747 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43831
I20260926 13:48:51.498116 814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43831 every 8 connection(s)
I20260926 13:48:51.498984 747 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:48:51.500188 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 747
I20260926 13:48:51.500319 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:48:51.500615 12969 external_mini_cluster.cc:1428] Setting key 1c368d62380f1be3b39f14ed9ebe225f
I20260926 13:48:51.502000 815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:51.508365 815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9d233f37c0842d4a1a0db66d78fe7db: Bootstrap starting.
Sep 26 13:48:51 dist-test-slave-hc1k krb5kdc[734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 13:48:51.510728 815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9d233f37c0842d4a1a0db66d78fe7db: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:51.511560 815 log.cc:824] T 00000000000000000000000000000000 P b9d233f37c0842d4a1a0db66d78fe7db: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:51.513571 815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9d233f37c0842d4a1a0db66d78fe7db: No bootstrap required, opened a new log
I20260926 13:48:51.515080 818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:51.501871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36605 (local address 127.0.0.1:43831)
0926 13:48:51.502264 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:51.502281 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:51.502321 (+ 40us) server_negotiation.cc:407] Connection header received
0926 13:48:51.503120 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:51.503141 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:51.503546 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:51.503928 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:51.504560 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:51.506036 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:51.506844 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:51.507534 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:51.507706 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:51.510167 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:51.510173 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:51.510188 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:51.510217 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:51.512103 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:51.512592 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:51.512598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:51.512603 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:51.512741 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:51.513047 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:51.513050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:51.513052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:51.513511 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:51.513731 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:51.513989 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 13:48:51.514201 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":118,"threads_started":1}
I20260926 13:48:51.517664 815 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9d233f37c0842d4a1a0db66d78fe7db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d233f37c0842d4a1a0db66d78fe7db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43831 } }
I20260926 13:48:51.517925 815 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9d233f37c0842d4a1a0db66d78fe7db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:51.517987 815 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9d233f37c0842d4a1a0db66d78fe7db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d233f37c0842d4a1a0db66d78fe7db, State: Initialized, Role: FOLLOWER
I20260926 13:48:51.518004 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:48:51.518391 815 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9d233f37c0842d4a1a0db66d78fe7db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d233f37c0842d4a1a0db66d78fe7db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43831 } }
I20260926 13:48:51.518515 815 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9d233f37c0842d4a1a0db66d78fe7db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:48:51.518575 815 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9d233f37c0842d4a1a0db66d78fe7db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:48:51.518664 815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9d233f37c0842d4a1a0db66d78fe7db [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:51.519521 815 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9d233f37c0842d4a1a0db66d78fe7db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d233f37c0842d4a1a0db66d78fe7db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43831 } }
I20260926 13:48:51.519994 815 leader_election.cc:302] T 00000000000000000000000000000000 P b9d233f37c0842d4a1a0db66d78fe7db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9d233f37c0842d4a1a0db66d78fe7db; no voters:
I20260926 13:48:51.520298 815 leader_election.cc:288] T 00000000000000000000000000000000 P b9d233f37c0842d4a1a0db66d78fe7db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:48:51.520375 825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9d233f37c0842d4a1a0db66d78fe7db [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:48:51.520586 825 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9d233f37c0842d4a1a0db66d78fe7db [term 1 LEADER]: Becoming Leader. State: Replica: b9d233f37c0842d4a1a0db66d78fe7db, State: Running, Role: LEADER
I20260926 13:48:51.520922 825 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9d233f37c0842d4a1a0db66d78fe7db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d233f37c0842d4a1a0db66d78fe7db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43831 } }
I20260926 13:48:51.521551 815 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9d233f37c0842d4a1a0db66d78fe7db [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:48:51.522513 826 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d233f37c0842d4a1a0db66d78fe7db [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9d233f37c0842d4a1a0db66d78fe7db. Latest consensus state: current_term: 1 leader_uuid: "b9d233f37c0842d4a1a0db66d78fe7db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d233f37c0842d4a1a0db66d78fe7db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43831 } } }
I20260926 13:48:51.522627 826 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d233f37c0842d4a1a0db66d78fe7db [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:51.522996 827 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d233f37c0842d4a1a0db66d78fe7db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9d233f37c0842d4a1a0db66d78fe7db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d233f37c0842d4a1a0db66d78fe7db" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43831 } } }
I20260926 13:48:51.523159 827 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d233f37c0842d4a1a0db66d78fe7db [sys.catalog]: This master's current role is: LEADER
W20260926 13:48:51.526911 841 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b9d233f37c0842d4a1a0db66d78fe7db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:48:51.526994 841 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:48:51.527064 834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:48:51.527691 818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:51.519883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59315 (local address 127.0.0.1:43831)
0926 13:48:51.519982 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:51.519986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:51.520120 (+ 134us) server_negotiation.cc:407] Connection header received
0926 13:48:51.520243 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:51.520246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:51.520280 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:51.520347 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:51.521048 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:51.521602 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:51.522582 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:51.522911 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:51.522981 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:51.524501 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:51.524505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:51.524508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:51.524530 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:51.525891 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:51.526405 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:51.526409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:51.526411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:51.526496 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:51.526815 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:51.526818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:51.526820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:51.527018 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:51.527159 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:51.527422 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 13:48:51.527509 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":34}
I20260926 13:48:51.528213 834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:48:51.533490 834 catalog_manager.cc:1347] Generated new cluster ID: 5c101a0814b44d35aaa97a1852010813
I20260926 13:48:51.533547 834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:48:51.559762 834 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:48:51.561025 834 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:48:51.569891 834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9d233f37c0842d4a1a0db66d78fe7db: Generated new TSK 0
I20260926 13:48:51.570556 834 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:48:51.584692 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 747
W20260926 13:48:51.589320 732 connection.cc:535] client connection to 127.0.0.1:43831 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43831): Connection reset by peer (error 104)
2026-09-26T13:48:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:48:51 dist-test-slave-hc1k krb5kdc[846](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:48:51 dist-test-slave-hc1k krb5kdc[846](info): set up 2 sockets
Sep 26 13:48:51 dist-test-slave-hc1k krb5kdc[846](info): commencing operation
krb5kdc: starting...
W20260926 13:48:52.527356 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:48:52 dist-test-slave-hc1k krb5kdc[846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:48:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:48:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:48:52.633944 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40307
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:48:52.734711 859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:52.734946 859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:52.735001 859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:52.738926 859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:48:52.738979 859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:52.739004 859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:52.739024 859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:48:52.739053 859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:48:52.744019 859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40307
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:52.744887 859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:52.745806 859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:52.753456 865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:52.753443 864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:52.753439 867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:52.753510 859 server_base.cc:1034] running on GCE node
I20260926 13:48:52.754130 859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:52.755101 859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:52.756255 859 hybrid_clock.cc:648] HybridClock initialized: now 1790430532756235 us; error 28 us; skew 500 ppm
Sep 26 13:48:52 dist-test-slave-hc1k krb5kdc[846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:52.758728 859 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:48:52.759467 859 webserver.cc:460] Webserver started at http://127.0.0.1:39687/ using document root <none> and password file <none>
I20260926 13:48:52.759970 859 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:52.760025 859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:52.760176 859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:52.762032 859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "520eeb2391844a7eb9cdcf7b2f03a5d6"
format_stamp: "Formatted at 2026-09-26 13:48:52 on dist-test-slave-hc1k"
server_key: "3b08fe788a10543dc87b71c157063921"
server_key_iv: "e186161d8522784a682c2c1e228c5dc8"
server_key_version: "encryptionkey@0"
I20260926 13:48:52.762497 859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "520eeb2391844a7eb9cdcf7b2f03a5d6"
format_stamp: "Formatted at 2026-09-26 13:48:52 on dist-test-slave-hc1k"
server_key: "3b08fe788a10543dc87b71c157063921"
server_key_iv: "e186161d8522784a682c2c1e228c5dc8"
server_key_version: "encryptionkey@0"
I20260926 13:48:52.765762 859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:48:52.767926 874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:52.769268 859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 13:48:52.769397 859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "520eeb2391844a7eb9cdcf7b2f03a5d6"
format_stamp: "Formatted at 2026-09-26 13:48:52 on dist-test-slave-hc1k"
server_key: "3b08fe788a10543dc87b71c157063921"
server_key_iv: "e186161d8522784a682c2c1e228c5dc8"
server_key_version: "encryptionkey@0"
I20260926 13:48:52.769496 859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:52.789819 859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:52.793283 859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:52.793519 859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:52.802057 926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40307 every 8 connection(s)
I20260926 13:48:52.802054 859 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40307
I20260926 13:48:52.803411 859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:48:52.806654 927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:52.808525 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 859
I20260926 13:48:52.808651 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:48:52.808923 12969 external_mini_cluster.cc:1428] Setting key 1122d452a03a7e17e2515beb7d2c130b
I20260926 13:48:52.816412 927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 520eeb2391844a7eb9cdcf7b2f03a5d6: Bootstrap starting.
I20260926 13:48:52.818753 927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 520eeb2391844a7eb9cdcf7b2f03a5d6: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:52.819543 927 log.cc:824] T 00000000000000000000000000000000 P 520eeb2391844a7eb9cdcf7b2f03a5d6: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:52.820785 930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:52.810560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40505 (local address 127.0.0.1:40307)
0926 13:48:52.810970 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:52.810988 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:52.811980 (+ 992us) server_negotiation.cc:407] Connection header received
0926 13:48:52.812795 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:52.812817 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:52.813235 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:52.813589 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:52.815098 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:52.816580 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:52.817425 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:52.818067 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:52.818214 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:52.818966 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:52.818972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:52.818988 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:48:52.819451 (+ 463us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:52.819520 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:48:52.819548 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:52.819714 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:52.819870 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 13:48:52.819999 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":131,"threads_started":1}
I20260926 13:48:52.822137 927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 520eeb2391844a7eb9cdcf7b2f03a5d6: No bootstrap required, opened a new log
I20260926 13:48:52.823344 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:48:52.825737 927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 520eeb2391844a7eb9cdcf7b2f03a5d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520eeb2391844a7eb9cdcf7b2f03a5d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40307 } }
I20260926 13:48:52.826004 927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 520eeb2391844a7eb9cdcf7b2f03a5d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:52.826058 927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 520eeb2391844a7eb9cdcf7b2f03a5d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 520eeb2391844a7eb9cdcf7b2f03a5d6, State: Initialized, Role: FOLLOWER
I20260926 13:48:52.826432 927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 520eeb2391844a7eb9cdcf7b2f03a5d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520eeb2391844a7eb9cdcf7b2f03a5d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40307 } }
I20260926 13:48:52.826540 927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 520eeb2391844a7eb9cdcf7b2f03a5d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:48:52.826596 927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 520eeb2391844a7eb9cdcf7b2f03a5d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:48:52.826683 927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 520eeb2391844a7eb9cdcf7b2f03a5d6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:52.827443 927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 520eeb2391844a7eb9cdcf7b2f03a5d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520eeb2391844a7eb9cdcf7b2f03a5d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40307 } }
I20260926 13:48:52.827772 927 leader_election.cc:302] T 00000000000000000000000000000000 P 520eeb2391844a7eb9cdcf7b2f03a5d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 520eeb2391844a7eb9cdcf7b2f03a5d6; no voters:
I20260926 13:48:52.828035 927 leader_election.cc:288] T 00000000000000000000000000000000 P 520eeb2391844a7eb9cdcf7b2f03a5d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:48:52.828152 937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 520eeb2391844a7eb9cdcf7b2f03a5d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:48:52.828787 937 raft_consensus.cc:686] T 00000000000000000000000000000000 P 520eeb2391844a7eb9cdcf7b2f03a5d6 [term 1 LEADER]: Becoming Leader. State: Replica: 520eeb2391844a7eb9cdcf7b2f03a5d6, State: Running, Role: LEADER
I20260926 13:48:52.828934 930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:52.825142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53769 (local address 127.0.0.1:40307)
0926 13:48:52.825235 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:52.825239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:52.825473 (+ 234us) server_negotiation.cc:407] Connection header received
0926 13:48:52.825572 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:52.825574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:52.825608 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:52.825665 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:52.825914 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:52.826507 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:52.827340 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:52.827600 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:52.827707 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:52.828494 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:52.828497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:52.828500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:48:52.828510 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:52.828536 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:48:52.828547 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:52.828622 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:52.828786 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 13:48:52.828817 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 13:48:52.829108 927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 520eeb2391844a7eb9cdcf7b2f03a5d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:48:52.829113 937 consensus_queue.cc:237] T 00000000000000000000000000000000 P 520eeb2391844a7eb9cdcf7b2f03a5d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520eeb2391844a7eb9cdcf7b2f03a5d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40307 } }
I20260926 13:48:52.831091 939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 520eeb2391844a7eb9cdcf7b2f03a5d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 520eeb2391844a7eb9cdcf7b2f03a5d6. Latest consensus state: current_term: 1 leader_uuid: "520eeb2391844a7eb9cdcf7b2f03a5d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520eeb2391844a7eb9cdcf7b2f03a5d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40307 } } }
I20260926 13:48:52.831075 938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 520eeb2391844a7eb9cdcf7b2f03a5d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "520eeb2391844a7eb9cdcf7b2f03a5d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "520eeb2391844a7eb9cdcf7b2f03a5d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40307 } } }
I20260926 13:48:52.831506 938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 520eeb2391844a7eb9cdcf7b2f03a5d6 [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:52.831204 939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 520eeb2391844a7eb9cdcf7b2f03a5d6 [sys.catalog]: This master's current role is: LEADER
W20260926 13:48:52.833791 951 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 520eeb2391844a7eb9cdcf7b2f03a5d6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:48:52.833861 951 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:48:52.833950 952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:48:52.834777 952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:48:52.839534 952 catalog_manager.cc:1347] Generated new cluster ID: 8507169f81304f8883572ed971e80908
I20260926 13:48:52.839593 952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:48:52.854322 952 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:48:52.855336 952 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:48:52.866948 952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 520eeb2391844a7eb9cdcf7b2f03a5d6: Generated new TSK 0
I20260926 13:48:52.867626 952 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:48:52.874358 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 859
W20260926 13:48:52.878969 844 connection.cc:535] client connection to 127.0.0.1:40307 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40307): Connection reset by peer (error 104)
2026-09-26T13:48:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1289 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:48:52 dist-test-slave-hc1k krb5kdc[958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:48:52 dist-test-slave-hc1k krb5kdc[958](info): set up 2 sockets
Sep 26 13:48:52 dist-test-slave-hc1k krb5kdc[958](info): commencing operation
krb5kdc: starting...
W20260926 13:48:53.815485 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:48:53 dist-test-slave-hc1k krb5kdc[958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:48:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:48:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:48:53.926183 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38703
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:48:54.027172 971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:54.027395 971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:54.027431 971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:54.031296 971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:48:54.031348 971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:54.031366 971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:54.031383 971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:48:54.031399 971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:48:54.036338 971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38703
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:54.037176 971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:54.038058 971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:54.045574 976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:54.045620 977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:54.045584 979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:54.045836 971 server_base.cc:1034] running on GCE node
I20260926 13:48:54.046388 971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:54.047369 971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:54.048517 971 hybrid_clock.cc:648] HybridClock initialized: now 1790430534048504 us; error 23 us; skew 500 ppm
Sep 26 13:48:54 dist-test-slave-hc1k krb5kdc[958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:54.050942 971 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:48:54.051707 971 webserver.cc:460] Webserver started at http://127.0.0.1:40593/ using document root <none> and password file <none>
I20260926 13:48:54.052182 971 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:54.052241 971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:54.052424 971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:54.054215 971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "cc21318b71464937907969a9e8a30b81"
format_stamp: "Formatted at 2026-09-26 13:48:54 on dist-test-slave-hc1k"
server_key: "770974330d36217e046c59c2509be20d"
server_key_iv: "5c70f623a09dabeae09132d1b794323f"
server_key_version: "encryptionkey@0"
I20260926 13:48:54.054706 971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "cc21318b71464937907969a9e8a30b81"
format_stamp: "Formatted at 2026-09-26 13:48:54 on dist-test-slave-hc1k"
server_key: "770974330d36217e046c59c2509be20d"
server_key_iv: "5c70f623a09dabeae09132d1b794323f"
server_key_version: "encryptionkey@0"
I20260926 13:48:54.058022 971 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 13:48:54.060263 986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:54.061344 971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:48:54.061445 971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "cc21318b71464937907969a9e8a30b81"
format_stamp: "Formatted at 2026-09-26 13:48:54 on dist-test-slave-hc1k"
server_key: "770974330d36217e046c59c2509be20d"
server_key_iv: "5c70f623a09dabeae09132d1b794323f"
server_key_version: "encryptionkey@0"
I20260926 13:48:54.061527 971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:54.065580 971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:54.068040 971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:54.068183 971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:54.074158 971 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38703
I20260926 13:48:54.074170 1038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38703 every 8 connection(s)
I20260926 13:48:54.074941 971 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:48:54.077426 1039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:54.080585 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 971
I20260926 13:48:54.080699 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:48:54.080967 12969 external_mini_cluster.cc:1428] Setting key 5d235e19271c0b542e4673e87ab1c827
I20260926 13:48:54.082777 1039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc21318b71464937907969a9e8a30b81: Bootstrap starting.
I20260926 13:48:54.085309 1039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc21318b71464937907969a9e8a30b81: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:54.085937 1042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:54.082142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51751 (local address 127.0.0.1:38703)
0926 13:48:54.082539 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:54.082555 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:54.082595 (+ 40us) server_negotiation.cc:407] Connection header received
0926 13:48:54.083359 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:54.083383 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:54.083748 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:54.084102 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:54.084678 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:54.084683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:54.084695 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:48:54.084705 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:54.084757 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:48:54.084781 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:54.084876 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:54.084938 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 13:48:54.085090 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":107,"threads_started":1}
I20260926 13:48:54.086134 1039 log.cc:824] T 00000000000000000000000000000000 P cc21318b71464937907969a9e8a30b81: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:54.088037 1039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc21318b71464937907969a9e8a30b81: No bootstrap required, opened a new log
I20260926 13:48:54.088464 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:48:54.091449 1042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:54.090212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41513 (local address 127.0.0.1:38703)
0926 13:48:54.090319 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:54.090323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:54.090422 (+ 99us) server_negotiation.cc:407] Connection header received
0926 13:48:54.090495 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:54.090498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:54.090531 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:54.090621 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:54.091143 (+ 522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:54.091146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:54.091149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:48:54.091160 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:54.091184 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:48:54.091195 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:54.091288 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:54.091317 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 13:48:54.091348 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 13:48:54.091616 1039 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc21318b71464937907969a9e8a30b81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc21318b71464937907969a9e8a30b81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38703 } }
I20260926 13:48:54.091879 1039 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc21318b71464937907969a9e8a30b81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:54.091938 1039 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc21318b71464937907969a9e8a30b81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc21318b71464937907969a9e8a30b81, State: Initialized, Role: FOLLOWER
I20260926 13:48:54.092319 1039 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc21318b71464937907969a9e8a30b81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc21318b71464937907969a9e8a30b81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38703 } }
I20260926 13:48:54.092435 1039 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc21318b71464937907969a9e8a30b81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:48:54.092479 1039 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc21318b71464937907969a9e8a30b81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:48:54.092566 1039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc21318b71464937907969a9e8a30b81 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:54.093273 1039 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc21318b71464937907969a9e8a30b81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc21318b71464937907969a9e8a30b81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38703 } }
I20260926 13:48:54.093535 1039 leader_election.cc:302] T 00000000000000000000000000000000 P cc21318b71464937907969a9e8a30b81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc21318b71464937907969a9e8a30b81; no voters:
I20260926 13:48:54.093794 1039 leader_election.cc:288] T 00000000000000000000000000000000 P cc21318b71464937907969a9e8a30b81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:48:54.093863 1049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc21318b71464937907969a9e8a30b81 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:48:54.094043 1049 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc21318b71464937907969a9e8a30b81 [term 1 LEADER]: Becoming Leader. State: Replica: cc21318b71464937907969a9e8a30b81, State: Running, Role: LEADER
I20260926 13:48:54.094340 1049 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc21318b71464937907969a9e8a30b81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc21318b71464937907969a9e8a30b81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38703 } }
I20260926 13:48:54.094806 1039 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc21318b71464937907969a9e8a30b81 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:48:54.095875 1051 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc21318b71464937907969a9e8a30b81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc21318b71464937907969a9e8a30b81. Latest consensus state: current_term: 1 leader_uuid: "cc21318b71464937907969a9e8a30b81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc21318b71464937907969a9e8a30b81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38703 } } }
I20260926 13:48:54.095993 1051 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc21318b71464937907969a9e8a30b81 [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:54.096271 1050 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc21318b71464937907969a9e8a30b81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc21318b71464937907969a9e8a30b81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc21318b71464937907969a9e8a30b81" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38703 } } }
I20260926 13:48:54.096329 1058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:48:54.099510 1058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:48:54.100979 1050 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc21318b71464937907969a9e8a30b81 [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:54.104028 1058 catalog_manager.cc:1347] Generated new cluster ID: 151f5b432a3d4f20abdb342c620270a2
I20260926 13:48:54.104084 1058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:48:54.114117 1058 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:48:54.114368 1058 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:48:54.122952 1058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc21318b71464937907969a9e8a30b81: Generated new TSK 0
I20260926 13:48:54.123484 1058 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:48:54.127743 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 971
2026-09-26T13:48:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1253 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:48:54 dist-test-slave-hc1k krb5kdc[1070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:48:54 dist-test-slave-hc1k krb5kdc[1070](info): set up 2 sockets
Sep 26 13:48:54 dist-test-slave-hc1k krb5kdc[1070](info): commencing operation
krb5kdc: starting...
W20260926 13:48:55.075408 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:48:55 dist-test-slave-hc1k krb5kdc[1070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:48:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:48:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:48:55.182498 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40921
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:48:55.285028 1083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:55.285266 1083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:55.285311 1083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:55.289218 1083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:48:55.289275 1083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:55.289297 1083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:55.289314 1083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:48:55.289335 1083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:48:55.294312 1083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:55.295184 1083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:55.296139 1083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:55.303736 1088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:55.303819 1091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:55.303881 1083 server_base.cc:1034] running on GCE node
W20260926 13:48:55.303737 1089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:55.304477 1083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:55.305433 1083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:55.306612 1083 hybrid_clock.cc:648] HybridClock initialized: now 1790430535306596 us; error 25 us; skew 500 ppm
Sep 26 13:48:55 dist-test-slave-hc1k krb5kdc[1070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:55.309252 1083 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:48:55.310019 1083 webserver.cc:460] Webserver started at http://127.0.0.1:34411/ using document root <none> and password file <none>
I20260926 13:48:55.310602 1083 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:55.310667 1083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:55.310819 1083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:55.312650 1083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "f7dfb31cf4774116b593e965024e5033"
format_stamp: "Formatted at 2026-09-26 13:48:55 on dist-test-slave-hc1k"
server_key: "a14708488b3c03605351336a665ecffd"
server_key_iv: "2b6794f024f58f7d7528a3be7252b80a"
server_key_version: "encryptionkey@0"
I20260926 13:48:55.313104 1083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "f7dfb31cf4774116b593e965024e5033"
format_stamp: "Formatted at 2026-09-26 13:48:55 on dist-test-slave-hc1k"
server_key: "a14708488b3c03605351336a665ecffd"
server_key_iv: "2b6794f024f58f7d7528a3be7252b80a"
server_key_version: "encryptionkey@0"
I20260926 13:48:55.316327 1083 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:48:55.318392 1098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:55.319461 1083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:48:55.319561 1083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "f7dfb31cf4774116b593e965024e5033"
format_stamp: "Formatted at 2026-09-26 13:48:55 on dist-test-slave-hc1k"
server_key: "a14708488b3c03605351336a665ecffd"
server_key_iv: "2b6794f024f58f7d7528a3be7252b80a"
server_key_version: "encryptionkey@0"
I20260926 13:48:55.319640 1083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:55.349231 1083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:55.351935 1083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:55.352102 1083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:55.358685 1083 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40921
I20260926 13:48:55.358741 1150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40921 every 8 connection(s)
I20260926 13:48:55.359639 1083 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:48:55.362264 1151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:55.367213 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 1083
I20260926 13:48:55.367339 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:48:55.367516 1151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7dfb31cf4774116b593e965024e5033: Bootstrap starting.
I20260926 13:48:55.367682 12969 external_mini_cluster.cc:1428] Setting key 8b6d2262a116294a797b19404c74e5d7
I20260926 13:48:55.369836 1151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7dfb31cf4774116b593e965024e5033: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:55.370509 1151 log.cc:824] T 00000000000000000000000000000000 P f7dfb31cf4774116b593e965024e5033: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:55.372206 1151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7dfb31cf4774116b593e965024e5033: No bootstrap required, opened a new log
I20260926 13:48:55.375464 1151 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7dfb31cf4774116b593e965024e5033 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7dfb31cf4774116b593e965024e5033" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40921 } }
I20260926 13:48:55.375686 1151 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7dfb31cf4774116b593e965024e5033 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:55.375741 1151 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7dfb31cf4774116b593e965024e5033 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7dfb31cf4774116b593e965024e5033, State: Initialized, Role: FOLLOWER
Sep 26 13:48:55 dist-test-slave-hc1k krb5kdc[1070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 13:48:55.376150 1151 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7dfb31cf4774116b593e965024e5033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7dfb31cf4774116b593e965024e5033" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40921 } }
I20260926 13:48:55.376312 1151 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7dfb31cf4774116b593e965024e5033 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:48:55.376377 1151 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7dfb31cf4774116b593e965024e5033 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:48:55.376470 1151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7dfb31cf4774116b593e965024e5033 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:55.377369 1151 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7dfb31cf4774116b593e965024e5033 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7dfb31cf4774116b593e965024e5033" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40921 } }
I20260926 13:48:55.377693 1151 leader_election.cc:302] T 00000000000000000000000000000000 P f7dfb31cf4774116b593e965024e5033 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7dfb31cf4774116b593e965024e5033; no voters:
I20260926 13:48:55.377969 1151 leader_election.cc:288] T 00000000000000000000000000000000 P f7dfb31cf4774116b593e965024e5033 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:48:55.378052 1156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7dfb31cf4774116b593e965024e5033 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:48:55.378202 1156 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7dfb31cf4774116b593e965024e5033 [term 1 LEADER]: Becoming Leader. State: Replica: f7dfb31cf4774116b593e965024e5033, State: Running, Role: LEADER
I20260926 13:48:55.378535 1156 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7dfb31cf4774116b593e965024e5033 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7dfb31cf4774116b593e965024e5033" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40921 } }
I20260926 13:48:55.379151 1151 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7dfb31cf4774116b593e965024e5033 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:48:55.379616 1156 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7dfb31cf4774116b593e965024e5033 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7dfb31cf4774116b593e965024e5033. Latest consensus state: current_term: 1 leader_uuid: "f7dfb31cf4774116b593e965024e5033" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7dfb31cf4774116b593e965024e5033" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40921 } } }
I20260926 13:48:55.380335 1156 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7dfb31cf4774116b593e965024e5033 [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:55.380224 1157 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7dfb31cf4774116b593e965024e5033 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7dfb31cf4774116b593e965024e5033" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7dfb31cf4774116b593e965024e5033" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40921 } } }
I20260926 13:48:55.380631 1157 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7dfb31cf4774116b593e965024e5033 [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:55.381585 1165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:48:55.381875 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:55.369015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55099 (local address 127.0.0.1:40921)
0926 13:48:55.369419 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:55.369435 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:55.369471 (+ 36us) server_negotiation.cc:407] Connection header received
0926 13:48:55.370102 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:55.370120 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:55.370463 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:55.370775 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:55.371507 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:55.372553 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:55.373375 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:55.373849 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:55.373993 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:55.376646 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:55.376651 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:55.376662 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:55.376688 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:55.378329 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:55.378780 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:55.378785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:55.378797 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:55.378903 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:55.379214 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:55.379217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:55.379219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:55.379683 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:55.380510 (+ 827us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:55.380648 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 13:48:55.380936 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":115,"threads_started":1}
I20260926 13:48:55.384649 1165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:48:55.384646 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:48:55.389901 1165 catalog_manager.cc:1347] Generated new cluster ID: 35e963a1c03e4ed4b539d4b8ceb2f2cf
I20260926 13:48:55.389953 1165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:48:55.395576 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:55.386689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46769 (local address 127.0.0.1:40921)
0926 13:48:55.386781 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:55.386785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:55.387942 (+ 1157us) server_negotiation.cc:407] Connection header received
0926 13:48:55.388040 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:55.388043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:55.388086 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:55.388165 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:55.388925 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:55.389602 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:55.390710 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:55.391057 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:55.391144 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:55.392667 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:55.392670 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:55.392673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:55.392694 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:55.394136 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:55.394559 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:55.394562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:55.394564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:55.394629 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:55.394859 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:55.394862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:55.394863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:55.395027 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:55.395142 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:55.395342 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 13:48:55.395438 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 13:48:55.401643 1165 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:48:55.402781 1165 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:48:55.408668 1165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7dfb31cf4774116b593e965024e5033: Generated new TSK 0
I20260926 13:48:55.409391 1165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:48:55.413316 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 1083
2026-09-26T13:48:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1285 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:48:55 dist-test-slave-hc1k krb5kdc[1182](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:48:55 dist-test-slave-hc1k krb5kdc[1182](info): set up 2 sockets
Sep 26 13:48:55 dist-test-slave-hc1k krb5kdc[1182](info): commencing operation
krb5kdc: starting...
W20260926 13:48:56.351511 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:48:56 dist-test-slave-hc1k krb5kdc[1182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:48:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:48:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:48:56.463043 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38213
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:48:56.567800 1195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:56.568056 1195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:56.568096 1195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:56.571979 1195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:48:56.572028 1195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:56.572045 1195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:56.572060 1195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:48:56.572073 1195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:48:56.577129 1195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:56.578049 1195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:56.578974 1195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:56.586895 1200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:56.586952 1195 server_base.cc:1034] running on GCE node
W20260926 13:48:56.586921 1203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:56.586905 1201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:56.587623 1195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:56.588717 1195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:56.589867 1195 hybrid_clock.cc:648] HybridClock initialized: now 1790430536589853 us; error 27 us; skew 500 ppm
Sep 26 13:48:56 dist-test-slave-hc1k krb5kdc[1182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:56.595319 1195 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:48:56.596222 1195 webserver.cc:460] Webserver started at http://127.0.0.1:36521/ using document root <none> and password file <none>
I20260926 13:48:56.596742 1195 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:56.596812 1195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:56.596993 1195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:56.599009 1195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "1e045eceb5be47d8b04d77729dd431be"
format_stamp: "Formatted at 2026-09-26 13:48:56 on dist-test-slave-hc1k"
server_key: "9681100fecbd178c9c9ce5a099a02e8d"
server_key_iv: "78c63da2d15661d6f8a98f854edc87f5"
server_key_version: "encryptionkey@0"
I20260926 13:48:56.599519 1195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "1e045eceb5be47d8b04d77729dd431be"
format_stamp: "Formatted at 2026-09-26 13:48:56 on dist-test-slave-hc1k"
server_key: "9681100fecbd178c9c9ce5a099a02e8d"
server_key_iv: "78c63da2d15661d6f8a98f854edc87f5"
server_key_version: "encryptionkey@0"
I20260926 13:48:56.603025 1195 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:48:56.605436 1210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:56.606597 1195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:48:56.606709 1195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "1e045eceb5be47d8b04d77729dd431be"
format_stamp: "Formatted at 2026-09-26 13:48:56 on dist-test-slave-hc1k"
server_key: "9681100fecbd178c9c9ce5a099a02e8d"
server_key_iv: "78c63da2d15661d6f8a98f854edc87f5"
server_key_version: "encryptionkey@0"
I20260926 13:48:56.606801 1195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:56.632529 1195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:56.635259 1195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:56.635426 1195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:56.641757 1195 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38213
I20260926 13:48:56.641810 1262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38213 every 8 connection(s)
I20260926 13:48:56.642637 1195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:48:56.645341 1263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:56.648968 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 1195
I20260926 13:48:56.649070 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:48:56.649323 12969 external_mini_cluster.cc:1428] Setting key bcab3a25c6973da6b6b6cf8ab38a04a7
I20260926 13:48:56.650851 1263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e045eceb5be47d8b04d77729dd431be: Bootstrap starting.
I20260926 13:48:56.653064 1263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e045eceb5be47d8b04d77729dd431be: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:56.653800 1263 log.cc:824] T 00000000000000000000000000000000 P 1e045eceb5be47d8b04d77729dd431be: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:56.655755 1263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e045eceb5be47d8b04d77729dd431be: No bootstrap required, opened a new log
I20260926 13:48:56.657701 1266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:56.650540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35309 (local address 127.0.0.1:38213)
0926 13:48:56.650917 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:56.650932 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:56.650968 (+ 36us) server_negotiation.cc:407] Connection header received
0926 13:48:56.651570 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:56.651586 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:56.651960 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:56.652219 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:56.652523 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:56.653496 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:56.654548 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:56.655005 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:56.655119 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:56.655930 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:56.655935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:56.655947 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:48:56.656315 (+ 368us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:56.656366 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:48:56.656388 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:56.656461 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:56.656609 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 13:48:56.656789 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":105,"threads_started":1}
I20260926 13:48:56.658689 1263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e045eceb5be47d8b04d77729dd431be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e045eceb5be47d8b04d77729dd431be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38213 } }
I20260926 13:48:56.658936 1263 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e045eceb5be47d8b04d77729dd431be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:56.658987 1263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e045eceb5be47d8b04d77729dd431be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e045eceb5be47d8b04d77729dd431be, State: Initialized, Role: FOLLOWER
I20260926 13:48:56.659401 1263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e045eceb5be47d8b04d77729dd431be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e045eceb5be47d8b04d77729dd431be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38213 } }
I20260926 13:48:56.659572 1263 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e045eceb5be47d8b04d77729dd431be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:48:56.659642 1263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e045eceb5be47d8b04d77729dd431be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:48:56.659758 1263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e045eceb5be47d8b04d77729dd431be [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:56.660457 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:48:56.660717 1263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e045eceb5be47d8b04d77729dd431be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e045eceb5be47d8b04d77729dd431be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38213 } }
I20260926 13:48:56.661016 1263 leader_election.cc:302] T 00000000000000000000000000000000 P 1e045eceb5be47d8b04d77729dd431be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e045eceb5be47d8b04d77729dd431be; no voters:
I20260926 13:48:56.661274 1263 leader_election.cc:288] T 00000000000000000000000000000000 P 1e045eceb5be47d8b04d77729dd431be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:48:56.661396 1268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e045eceb5be47d8b04d77729dd431be [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:48:56.661608 1268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e045eceb5be47d8b04d77729dd431be [term 1 LEADER]: Becoming Leader. State: Replica: 1e045eceb5be47d8b04d77729dd431be, State: Running, Role: LEADER
I20260926 13:48:56.661988 1268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e045eceb5be47d8b04d77729dd431be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e045eceb5be47d8b04d77729dd431be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38213 } }
I20260926 13:48:56.662390 1263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e045eceb5be47d8b04d77729dd431be [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:48:56.663161 1272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e045eceb5be47d8b04d77729dd431be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e045eceb5be47d8b04d77729dd431be. Latest consensus state: current_term: 1 leader_uuid: "1e045eceb5be47d8b04d77729dd431be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e045eceb5be47d8b04d77729dd431be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38213 } } }
I20260926 13:48:56.663300 1272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e045eceb5be47d8b04d77729dd431be [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:56.663522 1269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e045eceb5be47d8b04d77729dd431be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e045eceb5be47d8b04d77729dd431be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e045eceb5be47d8b04d77729dd431be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38213 } } }
I20260926 13:48:56.663604 1269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e045eceb5be47d8b04d77729dd431be [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:56.667078 1266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:56.662468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54589 (local address 127.0.0.1:38213)
0926 13:48:56.662572 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:56.662576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:56.662926 (+ 350us) server_negotiation.cc:407] Connection header received
0926 13:48:56.663072 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:56.663075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:56.663120 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:56.663170 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:56.663569 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:56.664328 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:56.665410 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:56.665729 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:56.665835 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:56.666674 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:56.666677 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:56.666680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:48:56.666692 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:56.666719 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:48:56.666733 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:56.666810 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:56.666908 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 13:48:56.666958 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 13:48:56.667239 1288 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e045eceb5be47d8b04d77729dd431be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:48:56.667328 1288 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:48:56.667416 1282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:48:56.668172 1282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:48:56.672948 1282 catalog_manager.cc:1347] Generated new cluster ID: d1b5208b3f374922889e8723b16ff0b0
I20260926 13:48:56.673013 1282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:48:56.699080 1282 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:48:56.700158 1282 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:48:56.711624 1282 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e045eceb5be47d8b04d77729dd431be: Generated new TSK 0
I20260926 13:48:56.712544 1282 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:48:56.716674 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 1195
2026-09-26T13:48:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:48:56 dist-test-slave-hc1k krb5kdc[1294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:48:56 dist-test-slave-hc1k krb5kdc[1294](info): set up 2 sockets
Sep 26 13:48:56 dist-test-slave-hc1k krb5kdc[1294](info): commencing operation
krb5kdc: starting...
W20260926 13:48:57.659583 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:48:57 dist-test-slave-hc1k krb5kdc[1294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:48:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:48:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:48:57.766234 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42583
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:48:57.868232 1307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:57.868453 1307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:57.868492 1307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:57.872364 1307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:48:57.872412 1307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:57.872430 1307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:57.872444 1307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:48:57.872457 1307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:48:57.878154 1307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:57.879038 1307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:57.879972 1307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:57.887466 1312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:57.887477 1313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:57.887522 1315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:57.887616 1307 server_base.cc:1034] running on GCE node
I20260926 13:48:57.888193 1307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:57.889129 1307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:57.890280 1307 hybrid_clock.cc:648] HybridClock initialized: now 1790430537890270 us; error 37 us; skew 500 ppm
Sep 26 13:48:57 dist-test-slave-hc1k krb5kdc[1294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:57.892767 1307 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:48:57.893561 1307 webserver.cc:460] Webserver started at http://127.0.0.1:45327/ using document root <none> and password file <none>
I20260926 13:48:57.894038 1307 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:57.894093 1307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:57.894266 1307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:57.896160 1307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "bab23869e2184d859aa7a3683bca500f"
format_stamp: "Formatted at 2026-09-26 13:48:57 on dist-test-slave-hc1k"
server_key: "a53c9f2417219ac0a76103ad2f4c8993"
server_key_iv: "4692b98f5df36f44d95466246a61205d"
server_key_version: "encryptionkey@0"
I20260926 13:48:57.896648 1307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "bab23869e2184d859aa7a3683bca500f"
format_stamp: "Formatted at 2026-09-26 13:48:57 on dist-test-slave-hc1k"
server_key: "a53c9f2417219ac0a76103ad2f4c8993"
server_key_iv: "4692b98f5df36f44d95466246a61205d"
server_key_version: "encryptionkey@0"
I20260926 13:48:57.899900 1307 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:48:57.901993 1322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:57.903092 1307 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:48:57.903203 1307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "bab23869e2184d859aa7a3683bca500f"
format_stamp: "Formatted at 2026-09-26 13:48:57 on dist-test-slave-hc1k"
server_key: "a53c9f2417219ac0a76103ad2f4c8993"
server_key_iv: "4692b98f5df36f44d95466246a61205d"
server_key_version: "encryptionkey@0"
I20260926 13:48:57.903285 1307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:57.907382 1307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:57.910001 1307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:57.910147 1307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:57.916412 1307 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42583
I20260926 13:48:57.916438 1374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42583 every 8 connection(s)
I20260926 13:48:57.917222 1307 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:48:57.919744 1375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:57.920687 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 1307
I20260926 13:48:57.920807 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:48:57.921082 12969 external_mini_cluster.cc:1428] Setting key 8f16b50e3d0bb0ea8d4b29870566a3b9
I20260926 13:48:57.925355 1375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bab23869e2184d859aa7a3683bca500f: Bootstrap starting.
I20260926 13:48:57.925623 1378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:57.922213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58867 (local address 127.0.0.1:42583)
0926 13:48:57.922563 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:57.922578 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:57.922610 (+ 32us) server_negotiation.cc:407] Connection header received
0926 13:48:57.923209 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:57.923224 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:57.923497 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:57.923808 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:57.924513 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:57.924517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:57.924527 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:48:57.924536 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:57.924583 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:48:57.924600 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:57.924698 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:57.924800 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 13:48:57.924928 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":103,"threads_started":1}
I20260926 13:48:57.927580 1375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bab23869e2184d859aa7a3683bca500f: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:57.928237 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:48:57.928391 1375 log.cc:824] T 00000000000000000000000000000000 P bab23869e2184d859aa7a3683bca500f: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:57.930393 1375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bab23869e2184d859aa7a3683bca500f: No bootstrap required, opened a new log
I20260926 13:48:57.931636 1378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:57.930271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45391 (local address 127.0.0.1:42583)
0926 13:48:57.930376 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:57.930380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:57.930553 (+ 173us) server_negotiation.cc:407] Connection header received
0926 13:48:57.930641 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:57.930643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:57.930677 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:57.930758 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:57.931313 (+ 555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:57.931316 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:57.931319 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:48:57.931329 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:57.931356 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:48:57.931368 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:57.931447 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:57.931492 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 13:48:57.931526 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 13:48:57.934297 1375 raft_consensus.cc:348] T 00000000000000000000000000000000 P bab23869e2184d859aa7a3683bca500f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab23869e2184d859aa7a3683bca500f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42583 } }
I20260926 13:48:57.934499 1375 raft_consensus.cc:374] T 00000000000000000000000000000000 P bab23869e2184d859aa7a3683bca500f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:57.934552 1375 raft_consensus.cc:729] T 00000000000000000000000000000000 P bab23869e2184d859aa7a3683bca500f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bab23869e2184d859aa7a3683bca500f, State: Initialized, Role: FOLLOWER
I20260926 13:48:57.934929 1375 consensus_queue.cc:260] T 00000000000000000000000000000000 P bab23869e2184d859aa7a3683bca500f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab23869e2184d859aa7a3683bca500f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42583 } }
I20260926 13:48:57.935050 1375 raft_consensus.cc:388] T 00000000000000000000000000000000 P bab23869e2184d859aa7a3683bca500f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:48:57.935107 1375 raft_consensus.cc:482] T 00000000000000000000000000000000 P bab23869e2184d859aa7a3683bca500f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:48:57.935195 1375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bab23869e2184d859aa7a3683bca500f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:57.936048 1375 raft_consensus.cc:504] T 00000000000000000000000000000000 P bab23869e2184d859aa7a3683bca500f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab23869e2184d859aa7a3683bca500f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42583 } }
I20260926 13:48:57.936353 1375 leader_election.cc:302] T 00000000000000000000000000000000 P bab23869e2184d859aa7a3683bca500f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bab23869e2184d859aa7a3683bca500f; no voters:
I20260926 13:48:57.936602 1375 leader_election.cc:288] T 00000000000000000000000000000000 P bab23869e2184d859aa7a3683bca500f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:48:57.936704 1385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bab23869e2184d859aa7a3683bca500f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:48:57.936882 1385 raft_consensus.cc:686] T 00000000000000000000000000000000 P bab23869e2184d859aa7a3683bca500f [term 1 LEADER]: Becoming Leader. State: Replica: bab23869e2184d859aa7a3683bca500f, State: Running, Role: LEADER
I20260926 13:48:57.937211 1385 consensus_queue.cc:237] T 00000000000000000000000000000000 P bab23869e2184d859aa7a3683bca500f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab23869e2184d859aa7a3683bca500f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42583 } }
I20260926 13:48:57.937685 1375 sys_catalog.cc:564] T 00000000000000000000000000000000 P bab23869e2184d859aa7a3683bca500f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:48:57.938721 1387 sys_catalog.cc:455] T 00000000000000000000000000000000 P bab23869e2184d859aa7a3683bca500f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bab23869e2184d859aa7a3683bca500f. Latest consensus state: current_term: 1 leader_uuid: "bab23869e2184d859aa7a3683bca500f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab23869e2184d859aa7a3683bca500f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42583 } } }
I20260926 13:48:57.938760 1386 sys_catalog.cc:455] T 00000000000000000000000000000000 P bab23869e2184d859aa7a3683bca500f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bab23869e2184d859aa7a3683bca500f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bab23869e2184d859aa7a3683bca500f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42583 } } }
I20260926 13:48:57.938854 1387 sys_catalog.cc:458] T 00000000000000000000000000000000 P bab23869e2184d859aa7a3683bca500f [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:57.938872 1386 sys_catalog.cc:458] T 00000000000000000000000000000000 P bab23869e2184d859aa7a3683bca500f [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:57.939241 1394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:48:57.941812 1394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:48:57.946412 1394 catalog_manager.cc:1347] Generated new cluster ID: d3d395374bb44aefbbd9ea0e95cd3446
I20260926 13:48:57.946470 1394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:48:57.961321 1394 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:48:57.961607 1394 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:48:57.968670 1394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bab23869e2184d859aa7a3683bca500f: Generated new TSK 0
I20260926 13:48:57.969323 1394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:48:57.972544 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 1307
2026-09-26T13:48:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1255 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:48:58 dist-test-slave-hc1k krb5kdc[1406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:48:58 dist-test-slave-hc1k krb5kdc[1406](info): set up 2 sockets
Sep 26 13:48:58 dist-test-slave-hc1k krb5kdc[1406](info): commencing operation
krb5kdc: starting...
W20260926 13:48:58.919517 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:48:58 dist-test-slave-hc1k krb5kdc[1406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:48:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:48:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:48:59.026427 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46049
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:48:59.129352 1419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:48:59.129570 1419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:48:59.129606 1419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:48:59.133479 1419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:48:59.133525 1419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:48:59.133543 1419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:48:59.133556 1419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:48:59.133570 1419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:48:59.138438 1419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:48:59.139297 1419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:48:59.140192 1419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:48:59.147753 1425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:59.147841 1419 server_base.cc:1034] running on GCE node
W20260926 13:48:59.147779 1424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:48:59.147791 1427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:48:59.148584 1419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:48:59.149540 1419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:48:59.150657 1419 hybrid_clock.cc:648] HybridClock initialized: now 1790430539150643 us; error 24 us; skew 500 ppm
Sep 26 13:48:59 dist-test-slave-hc1k krb5kdc[1406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:48:59.153272 1419 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:48:59.154070 1419 webserver.cc:460] Webserver started at http://127.0.0.1:41275/ using document root <none> and password file <none>
I20260926 13:48:59.154575 1419 fs_manager.cc:359] Metadata directory not provided
I20260926 13:48:59.154636 1419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:48:59.154803 1419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:48:59.156716 1419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "4435be6c2a894501bbd31827dc5669e0"
format_stamp: "Formatted at 2026-09-26 13:48:59 on dist-test-slave-hc1k"
server_key: "a2c1e923306a9e65268be81cb9c91fe9"
server_key_iv: "0d862dc21ba18f9d88093eecbbaa8e26"
server_key_version: "encryptionkey@0"
I20260926 13:48:59.157215 1419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "4435be6c2a894501bbd31827dc5669e0"
format_stamp: "Formatted at 2026-09-26 13:48:59 on dist-test-slave-hc1k"
server_key: "a2c1e923306a9e65268be81cb9c91fe9"
server_key_iv: "0d862dc21ba18f9d88093eecbbaa8e26"
server_key_version: "encryptionkey@0"
I20260926 13:48:59.160570 1419 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:48:59.162688 1434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:48:59.163823 1419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:48:59.163933 1419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "4435be6c2a894501bbd31827dc5669e0"
format_stamp: "Formatted at 2026-09-26 13:48:59 on dist-test-slave-hc1k"
server_key: "a2c1e923306a9e65268be81cb9c91fe9"
server_key_iv: "0d862dc21ba18f9d88093eecbbaa8e26"
server_key_version: "encryptionkey@0"
I20260926 13:48:59.164016 1419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:48:59.181041 1419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:48:59.183574 1419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:48:59.183743 1419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:48:59.190037 1419 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46049
I20260926 13:48:59.190050 1486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46049 every 8 connection(s)
I20260926 13:48:59.190969 1419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:48:59.191325 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 1419
I20260926 13:48:59.191454 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:48:59.191754 12969 external_mini_cluster.cc:1428] Setting key 88ebc3091a40b44f0ca1c23693e335c3
I20260926 13:48:59.193769 1487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:48:59.199733 1487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4435be6c2a894501bbd31827dc5669e0: Bootstrap starting.
Sep 26 13:48:59 dist-test-slave-hc1k krb5kdc[1406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 13:48:59.202145 1487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4435be6c2a894501bbd31827dc5669e0: Neither blocks nor log segments found. Creating new log.
I20260926 13:48:59.202953 1487 log.cc:824] T 00000000000000000000000000000000 P 4435be6c2a894501bbd31827dc5669e0: Log is configured to *not* fsync() on all Append() calls
I20260926 13:48:59.205137 1487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4435be6c2a894501bbd31827dc5669e0: No bootstrap required, opened a new log
I20260926 13:48:59.205632 1490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:59.192938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:55229 (local address 127.0.0.1:46049)
0926 13:48:59.193313 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:59.193330 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:59.193365 (+ 35us) server_negotiation.cc:407] Connection header received
0926 13:48:59.193994 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:59.194012 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:59.194352 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:59.194610 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:59.195421 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:59.196450 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:59.197580 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:59.198065 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:59.198190 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:59.200750 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:59.200757 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:59.200773 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:59.200804 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:59.202597 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:59.203050 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:59.203056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:59.203061 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:59.203187 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:59.203559 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:59.203562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:59.203563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:59.204051 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:59.204247 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:59.204525 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 13:48:59.204737 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":112,"threads_started":1}
I20260926 13:48:59.208389 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:48:59.209052 1487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4435be6c2a894501bbd31827dc5669e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4435be6c2a894501bbd31827dc5669e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46049 } }
I20260926 13:48:59.209254 1487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4435be6c2a894501bbd31827dc5669e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:48:59.209309 1487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4435be6c2a894501bbd31827dc5669e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4435be6c2a894501bbd31827dc5669e0, State: Initialized, Role: FOLLOWER
I20260926 13:48:59.209721 1487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4435be6c2a894501bbd31827dc5669e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4435be6c2a894501bbd31827dc5669e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46049 } }
I20260926 13:48:59.209857 1487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4435be6c2a894501bbd31827dc5669e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:48:59.209918 1487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4435be6c2a894501bbd31827dc5669e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:48:59.210007 1487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4435be6c2a894501bbd31827dc5669e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:48:59.210958 1487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4435be6c2a894501bbd31827dc5669e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4435be6c2a894501bbd31827dc5669e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46049 } }
I20260926 13:48:59.211258 1487 leader_election.cc:302] T 00000000000000000000000000000000 P 4435be6c2a894501bbd31827dc5669e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4435be6c2a894501bbd31827dc5669e0; no voters:
I20260926 13:48:59.211557 1487 leader_election.cc:288] T 00000000000000000000000000000000 P 4435be6c2a894501bbd31827dc5669e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:48:59.211632 1497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4435be6c2a894501bbd31827dc5669e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:48:59.211860 1497 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4435be6c2a894501bbd31827dc5669e0 [term 1 LEADER]: Becoming Leader. State: Replica: 4435be6c2a894501bbd31827dc5669e0, State: Running, Role: LEADER
I20260926 13:48:59.212154 1497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4435be6c2a894501bbd31827dc5669e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4435be6c2a894501bbd31827dc5669e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46049 } }
I20260926 13:48:59.212773 1487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4435be6c2a894501bbd31827dc5669e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:48:59.213511 1498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4435be6c2a894501bbd31827dc5669e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4435be6c2a894501bbd31827dc5669e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4435be6c2a894501bbd31827dc5669e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46049 } } }
I20260926 13:48:59.213675 1498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4435be6c2a894501bbd31827dc5669e0 [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:59.213925 1498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4435be6c2a894501bbd31827dc5669e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4435be6c2a894501bbd31827dc5669e0. Latest consensus state: current_term: 1 leader_uuid: "4435be6c2a894501bbd31827dc5669e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4435be6c2a894501bbd31827dc5669e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46049 } } }
I20260926 13:48:59.214054 1498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4435be6c2a894501bbd31827dc5669e0 [sys.catalog]: This master's current role is: LEADER
I20260926 13:48:59.214358 1506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:48:59.217624 1506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:48:59.219491 1490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:48:59.210174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:53163 (local address 127.0.0.1:46049)
0926 13:48:59.210277 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 13:48:59.210281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:48:59.210532 (+ 251us) server_negotiation.cc:407] Connection header received
0926 13:48:59.210617 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:48:59.210620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:48:59.210662 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:48:59.210739 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:48:59.211325 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:59.212074 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:59.213177 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:48:59.213519 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:48:59.213614 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:48:59.215272 (+ 1658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:48:59.215275 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:48:59.215278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:48:59.215299 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:48:59.216826 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:59.217435 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:59.217438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:59.217440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:59.217563 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:48:59.218310 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:48:59.218313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:48:59.218324 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:48:59.218528 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:48:59.218658 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 13:48:59.219242 (+ 584us) server_negotiation.cc:299] Negotiation successful
0926 13:48:59.219339 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 13:48:59.222869 1506 catalog_manager.cc:1347] Generated new cluster ID: a95b20550688457aa57f0dc90404e0cc
I20260926 13:48:59.222929 1506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:48:59.231739 1506 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:48:59.232595 1506 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:48:59.237488 1506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4435be6c2a894501bbd31827dc5669e0: Generated new TSK 0
I20260926 13:48:59.238161 1506 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:48:59.246260 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 1419
W20260926 13:48:59.250607 1404 connection.cc:535] client connection to 127.0.0.1:46049 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46049): Connection reset by peer (error 104)
2026-09-26T13:48:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1273 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:48:59 dist-test-slave-hc1k krb5kdc[1518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:48:59 dist-test-slave-hc1k krb5kdc[1518](info): set up 2 sockets
Sep 26 13:48:59 dist-test-slave-hc1k krb5kdc[1518](info): commencing operation
krb5kdc: starting...
W20260926 13:49:00.191541 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:49:00 dist-test-slave-hc1k krb5kdc[1518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430540, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:49:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:49:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:49:00.304777 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39247
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:49:00.407792 1531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:49:00.408020 1531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:49:00.408059 1531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:49:00.411938 1531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:49:00.411991 1531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:49:00.412012 1531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:49:00.412034 1531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:49:00.412065 1531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:49:00.416994 1531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:49:00.417856 1531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:49:00.418747 1531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:49:00.426347 1539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:49:00.426533 1531 server_base.cc:1034] running on GCE node
W20260926 13:49:00.426343 1536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:00.426427 1537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:49:00.427189 1531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:49:00.428210 1531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:49:00.429416 1531 hybrid_clock.cc:648] HybridClock initialized: now 1790430540429393 us; error 39 us; skew 500 ppm
Sep 26 13:49:00 dist-test-slave-hc1k krb5kdc[1518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:49:00.431898 1531 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:49:00.432688 1531 webserver.cc:460] Webserver started at http://127.0.0.1:43473/ using document root <none> and password file <none>
I20260926 13:49:00.433171 1531 fs_manager.cc:359] Metadata directory not provided
I20260926 13:49:00.433245 1531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:49:00.433441 1531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:49:00.435283 1531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "12af4c13d1af467e9beccdc0852e3b62"
format_stamp: "Formatted at 2026-09-26 13:49:00 on dist-test-slave-hc1k"
server_key: "2186e07c2073a2bd67782d44c89260a3"
server_key_iv: "2ab6b6e75e47d23fa9675ea38ff9784e"
server_key_version: "encryptionkey@0"
I20260926 13:49:00.435843 1531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "12af4c13d1af467e9beccdc0852e3b62"
format_stamp: "Formatted at 2026-09-26 13:49:00 on dist-test-slave-hc1k"
server_key: "2186e07c2073a2bd67782d44c89260a3"
server_key_iv: "2ab6b6e75e47d23fa9675ea38ff9784e"
server_key_version: "encryptionkey@0"
I20260926 13:49:00.439173 1531 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 13:49:00.441412 1546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:49:00.442515 1531 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:49:00.442624 1531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "12af4c13d1af467e9beccdc0852e3b62"
format_stamp: "Formatted at 2026-09-26 13:49:00 on dist-test-slave-hc1k"
server_key: "2186e07c2073a2bd67782d44c89260a3"
server_key_iv: "2ab6b6e75e47d23fa9675ea38ff9784e"
server_key_version: "encryptionkey@0"
I20260926 13:49:00.442708 1531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:49:00.461433 1531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:49:00.463863 1531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:49:00.464010 1531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:49:00.470010 1531 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39247
I20260926 13:49:00.470054 1598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39247 every 8 connection(s)
I20260926 13:49:00.470813 1531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:49:00.473315 1599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:49:00.478168 1599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12af4c13d1af467e9beccdc0852e3b62: Bootstrap starting.
I20260926 13:49:00.479636 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 1531
I20260926 13:49:00.479784 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:49:00.480084 12969 external_mini_cluster.cc:1428] Setting key 0bacca560a5988974d52076ee2b84a89
I20260926 13:49:00.480549 1599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12af4c13d1af467e9beccdc0852e3b62: Neither blocks nor log segments found. Creating new log.
I20260926 13:49:00.481329 1599 log.cc:824] T 00000000000000000000000000000000 P 12af4c13d1af467e9beccdc0852e3b62: Log is configured to *not* fsync() on all Append() calls
I20260926 13:49:00.483080 1599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12af4c13d1af467e9beccdc0852e3b62: No bootstrap required, opened a new log
I20260926 13:49:00.486763 1599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12af4c13d1af467e9beccdc0852e3b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12af4c13d1af467e9beccdc0852e3b62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39247 } }
I20260926 13:49:00.486958 1599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12af4c13d1af467e9beccdc0852e3b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:49:00.486990 1599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12af4c13d1af467e9beccdc0852e3b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12af4c13d1af467e9beccdc0852e3b62, State: Initialized, Role: FOLLOWER
I20260926 13:49:00.487429 1599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12af4c13d1af467e9beccdc0852e3b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12af4c13d1af467e9beccdc0852e3b62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39247 } }
I20260926 13:49:00.487555 1599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12af4c13d1af467e9beccdc0852e3b62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:49:00.487609 1599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12af4c13d1af467e9beccdc0852e3b62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:49:00.487733 1599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12af4c13d1af467e9beccdc0852e3b62 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:49:00.488142 1602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:00.481057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:41245 (local address 127.0.0.1:39247)
0926 13:49:00.481420 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:00.481438 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:00.481475 (+ 37us) server_negotiation.cc:407] Connection header received
0926 13:49:00.482164 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:00.482182 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:00.482507 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:00.482750 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:00.483077 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:00.484070 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:00.485259 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:00.485689 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:00.485797 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:00.486544 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:00.486549 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:00.486560 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:49:00.486887 (+ 327us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:00.486933 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:49:00.486960 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:00.487075 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:00.487248 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 13:49:00.487386 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":91,"threads_started":1}
I20260926 13:49:00.488514 1599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12af4c13d1af467e9beccdc0852e3b62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12af4c13d1af467e9beccdc0852e3b62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39247 } }
I20260926 13:49:00.488802 1599 leader_election.cc:302] T 00000000000000000000000000000000 P 12af4c13d1af467e9beccdc0852e3b62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12af4c13d1af467e9beccdc0852e3b62; no voters:
I20260926 13:49:00.489051 1599 leader_election.cc:288] T 00000000000000000000000000000000 P 12af4c13d1af467e9beccdc0852e3b62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:49:00.489121 1604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12af4c13d1af467e9beccdc0852e3b62 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:49:00.489274 1604 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12af4c13d1af467e9beccdc0852e3b62 [term 1 LEADER]: Becoming Leader. State: Replica: 12af4c13d1af467e9beccdc0852e3b62, State: Running, Role: LEADER
I20260926 13:49:00.489598 1604 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12af4c13d1af467e9beccdc0852e3b62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12af4c13d1af467e9beccdc0852e3b62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39247 } }
I20260926 13:49:00.490182 1599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12af4c13d1af467e9beccdc0852e3b62 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:49:00.490871 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:49:00.491015 1605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12af4c13d1af467e9beccdc0852e3b62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12af4c13d1af467e9beccdc0852e3b62. Latest consensus state: current_term: 1 leader_uuid: "12af4c13d1af467e9beccdc0852e3b62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12af4c13d1af467e9beccdc0852e3b62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39247 } } }
I20260926 13:49:00.491134 1605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12af4c13d1af467e9beccdc0852e3b62 [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:00.492762 1606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12af4c13d1af467e9beccdc0852e3b62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12af4c13d1af467e9beccdc0852e3b62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12af4c13d1af467e9beccdc0852e3b62" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39247 } } }
I20260926 13:49:00.492915 1606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12af4c13d1af467e9beccdc0852e3b62 [sys.catalog]: This master's current role is: LEADER
W20260926 13:49:00.494966 1623 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12af4c13d1af467e9beccdc0852e3b62: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:49:00.495038 1623 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:49:00.495100 1612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:49:00.495757 1612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:49:00.497829 1602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:00.492913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:36967 (local address 127.0.0.1:39247)
0926 13:49:00.493015 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:00.493019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:00.493328 (+ 309us) server_negotiation.cc:407] Connection header received
0926 13:49:00.493432 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:00.493434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:00.493467 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:00.493524 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:00.493796 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:00.494509 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:00.496182 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:00.496506 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:00.496610 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:00.497417 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:00.497419 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:00.497421 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:49:00.497431 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:00.497454 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:49:00.497463 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:00.497533 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:00.497669 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 13:49:00.497701 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 13:49:00.501195 1612 catalog_manager.cc:1347] Generated new cluster ID: 9cb1ab4583b64c58a26effeb5ca581f4
I20260926 13:49:00.501242 1612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:49:00.523902 1612 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:49:00.524663 1612 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:49:00.530925 1612 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12af4c13d1af467e9beccdc0852e3b62: Generated new TSK 0
I20260926 13:49:00.531471 1612 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:49:00.538614 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 1531
W20260926 13:49:00.543076 1516 connection.cc:535] client connection to 127.0.0.1:39247 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39247): Connection reset by peer (error 104)
2026-09-26T13:49:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:49:00 dist-test-slave-hc1k krb5kdc[1630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:49:00 dist-test-slave-hc1k krb5kdc[1630](info): set up 2 sockets
Sep 26 13:49:00 dist-test-slave-hc1k krb5kdc[1630](info): commencing operation
krb5kdc: starting...
W20260926 13:49:01.483444 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:49:01 dist-test-slave-hc1k krb5kdc[1630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:49:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:49:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:49:01.589876 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46437
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:49:01.690104 1643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:49:01.690338 1643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:49:01.690376 1643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:49:01.694188 1643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:49:01.694236 1643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:49:01.694254 1643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:49:01.694268 1643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:49:01.694281 1643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:49:01.699118 1643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:49:01.699955 1643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:49:01.700803 1643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:49:01.708417 1648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:01.708457 1649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:01.708467 1651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:49:01.708621 1643 server_base.cc:1034] running on GCE node
I20260926 13:49:01.709115 1643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:49:01.710023 1643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:49:01.711172 1643 hybrid_clock.cc:648] HybridClock initialized: now 1790430541711156 us; error 29 us; skew 500 ppm
Sep 26 13:49:01 dist-test-slave-hc1k krb5kdc[1630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:49:01.716322 1643 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:49:01.717095 1643 webserver.cc:460] Webserver started at http://127.0.0.1:38073/ using document root <none> and password file <none>
I20260926 13:49:01.717571 1643 fs_manager.cc:359] Metadata directory not provided
I20260926 13:49:01.717629 1643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:49:01.717792 1643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:49:01.719606 1643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "4df5bebcf3a64903bc67d1c1383372df"
format_stamp: "Formatted at 2026-09-26 13:49:01 on dist-test-slave-hc1k"
server_key: "198cdcde519984cbc747c6d76db40db8"
server_key_iv: "c1214b41e679a259f528d5df99df6722"
server_key_version: "encryptionkey@0"
I20260926 13:49:01.720144 1643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "4df5bebcf3a64903bc67d1c1383372df"
format_stamp: "Formatted at 2026-09-26 13:49:01 on dist-test-slave-hc1k"
server_key: "198cdcde519984cbc747c6d76db40db8"
server_key_iv: "c1214b41e679a259f528d5df99df6722"
server_key_version: "encryptionkey@0"
I20260926 13:49:01.723361 1643 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 13:49:01.725551 1658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:49:01.726665 1643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:49:01.726770 1643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "4df5bebcf3a64903bc67d1c1383372df"
format_stamp: "Formatted at 2026-09-26 13:49:01 on dist-test-slave-hc1k"
server_key: "198cdcde519984cbc747c6d76db40db8"
server_key_iv: "c1214b41e679a259f528d5df99df6722"
server_key_version: "encryptionkey@0"
I20260926 13:49:01.726861 1643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:49:01.730917 1643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:49:01.733420 1643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:49:01.733561 1643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:49:01.739621 1643 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46437
I20260926 13:49:01.739640 1710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46437 every 8 connection(s)
I20260926 13:49:01.740418 1643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:49:01.742774 1711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:49:01.744405 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 1643
I20260926 13:49:01.744540 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:49:01.744841 12969 external_mini_cluster.cc:1428] Setting key 33a6f6f47bb3aee1ed6decfd479e2792
I20260926 13:49:01.748620 1711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4df5bebcf3a64903bc67d1c1383372df: Bootstrap starting.
I20260926 13:49:01.749269 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:01.746003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:45289 (local address 127.0.0.1:46437)
0926 13:49:01.746371 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:01.746386 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:01.746419 (+ 33us) server_negotiation.cc:407] Connection header received
0926 13:49:01.747008 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:01.747022 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:01.747266 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:01.747531 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:01.748239 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:01.748243 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:01.748255 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:49:01.748264 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:01.748316 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:49:01.748333 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:01.748426 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:01.748516 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 13:49:01.748646 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":97,"threads_started":1}
I20260926 13:49:01.750753 1711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4df5bebcf3a64903bc67d1c1383372df: Neither blocks nor log segments found. Creating new log.
I20260926 13:49:01.751480 1711 log.cc:824] T 00000000000000000000000000000000 P 4df5bebcf3a64903bc67d1c1383372df: Log is configured to *not* fsync() on all Append() calls
I20260926 13:49:01.751783 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:49:01.753400 1711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4df5bebcf3a64903bc67d1c1383372df: No bootstrap required, opened a new log
I20260926 13:49:01.755134 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:01.753812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:54543 (local address 127.0.0.1:46437)
0926 13:49:01.753904 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:01.753908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:01.754124 (+ 216us) server_negotiation.cc:407] Connection header received
0926 13:49:01.754224 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:01.754226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:01.754251 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:01.754314 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:01.754844 (+ 530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:01.754847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:01.754848 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:49:01.754857 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:01.754876 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:49:01.754885 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:01.754944 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:01.755018 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 13:49:01.755045 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 13:49:01.757119 1711 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4df5bebcf3a64903bc67d1c1383372df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df5bebcf3a64903bc67d1c1383372df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46437 } }
I20260926 13:49:01.757306 1711 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4df5bebcf3a64903bc67d1c1383372df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:49:01.757359 1711 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4df5bebcf3a64903bc67d1c1383372df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4df5bebcf3a64903bc67d1c1383372df, State: Initialized, Role: FOLLOWER
I20260926 13:49:01.757692 1711 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4df5bebcf3a64903bc67d1c1383372df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df5bebcf3a64903bc67d1c1383372df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46437 } }
I20260926 13:49:01.757807 1711 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4df5bebcf3a64903bc67d1c1383372df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:49:01.757865 1711 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4df5bebcf3a64903bc67d1c1383372df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:49:01.757949 1711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4df5bebcf3a64903bc67d1c1383372df [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:49:01.758651 1711 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4df5bebcf3a64903bc67d1c1383372df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df5bebcf3a64903bc67d1c1383372df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46437 } }
I20260926 13:49:01.758934 1711 leader_election.cc:302] T 00000000000000000000000000000000 P 4df5bebcf3a64903bc67d1c1383372df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4df5bebcf3a64903bc67d1c1383372df; no voters:
I20260926 13:49:01.759166 1711 leader_election.cc:288] T 00000000000000000000000000000000 P 4df5bebcf3a64903bc67d1c1383372df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:49:01.759253 1721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4df5bebcf3a64903bc67d1c1383372df [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:49:01.759442 1721 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4df5bebcf3a64903bc67d1c1383372df [term 1 LEADER]: Becoming Leader. State: Replica: 4df5bebcf3a64903bc67d1c1383372df, State: Running, Role: LEADER
I20260926 13:49:01.759786 1721 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4df5bebcf3a64903bc67d1c1383372df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df5bebcf3a64903bc67d1c1383372df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46437 } }
I20260926 13:49:01.760182 1711 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4df5bebcf3a64903bc67d1c1383372df [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:49:01.761121 1723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4df5bebcf3a64903bc67d1c1383372df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4df5bebcf3a64903bc67d1c1383372df. Latest consensus state: current_term: 1 leader_uuid: "4df5bebcf3a64903bc67d1c1383372df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df5bebcf3a64903bc67d1c1383372df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46437 } } }
I20260926 13:49:01.761291 1723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4df5bebcf3a64903bc67d1c1383372df [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:01.761178 1722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4df5bebcf3a64903bc67d1c1383372df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4df5bebcf3a64903bc67d1c1383372df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4df5bebcf3a64903bc67d1c1383372df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46437 } } }
I20260926 13:49:01.761564 1722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4df5bebcf3a64903bc67d1c1383372df [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:01.761652 1730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:49:01.764159 1730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:49:01.768617 1730 catalog_manager.cc:1347] Generated new cluster ID: 61f6adc09e2e482c80753ee3f81ac6f7
I20260926 13:49:01.768673 1730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:49:01.784914 1730 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:49:01.785209 1730 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:49:01.794425 1730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4df5bebcf3a64903bc67d1c1383372df: Generated new TSK 0
I20260926 13:49:01.795205 1730 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:49:01.802395 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 1643
2026-09-26T13:49:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1263 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:49:01 dist-test-slave-hc1k krb5kdc[1742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:49:01 dist-test-slave-hc1k krb5kdc[1742](info): set up 2 sockets
Sep 26 13:49:01 dist-test-slave-hc1k krb5kdc[1742](info): commencing operation
krb5kdc: starting...
W20260926 13:49:02.747605 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:49:02 dist-test-slave-hc1k krb5kdc[1742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:49:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:49:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:49:02.864565 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45477
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:49:02.966288 1755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:49:02.966499 1755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:49:02.966537 1755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:49:02.970386 1755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:49:02.970433 1755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:49:02.970450 1755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:49:02.970464 1755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:49:02.970479 1755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:49:02.975486 1755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45477
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:49:02.976361 1755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:49:02.977218 1755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:49:02.984860 1761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:02.984884 1760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:02.984885 1763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:49:02.984948 1755 server_base.cc:1034] running on GCE node
I20260926 13:49:02.985572 1755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:49:02.986549 1755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:49:02.987717 1755 hybrid_clock.cc:648] HybridClock initialized: now 1790430542987700 us; error 31 us; skew 500 ppm
Sep 26 13:49:02 dist-test-slave-hc1k krb5kdc[1742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:49:02.990113 1755 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:49:02.990849 1755 webserver.cc:460] Webserver started at http://127.0.0.1:40659/ using document root <none> and password file <none>
I20260926 13:49:02.991315 1755 fs_manager.cc:359] Metadata directory not provided
I20260926 13:49:02.991375 1755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:49:02.991537 1755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:49:02.993372 1755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "46c77a9331e04b998eaf846d3b6c62e8"
format_stamp: "Formatted at 2026-09-26 13:49:02 on dist-test-slave-hc1k"
server_key: "f99ac4bcbe8e0ba07b97357d62d1c6a9"
server_key_iv: "1190fbe30111df0db790836993b324f8"
server_key_version: "encryptionkey@0"
I20260926 13:49:02.993873 1755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "46c77a9331e04b998eaf846d3b6c62e8"
format_stamp: "Formatted at 2026-09-26 13:49:02 on dist-test-slave-hc1k"
server_key: "f99ac4bcbe8e0ba07b97357d62d1c6a9"
server_key_iv: "1190fbe30111df0db790836993b324f8"
server_key_version: "encryptionkey@0"
I20260926 13:49:02.997116 1755 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:49:02.999209 1770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:49:03.000329 1755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:49:03.000433 1755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "46c77a9331e04b998eaf846d3b6c62e8"
format_stamp: "Formatted at 2026-09-26 13:49:02 on dist-test-slave-hc1k"
server_key: "f99ac4bcbe8e0ba07b97357d62d1c6a9"
server_key_iv: "1190fbe30111df0db790836993b324f8"
server_key_version: "encryptionkey@0"
I20260926 13:49:03.000516 1755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:49:03.022773 1755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:49:03.025377 1755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:49:03.025537 1755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:49:03.031602 1755 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45477
I20260926 13:49:03.031668 1822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45477 every 8 connection(s)
I20260926 13:49:03.032449 1755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:49:03.034940 1823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:49:03.039188 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 1755
I20260926 13:49:03.039323 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:49:03.039572 12969 external_mini_cluster.cc:1428] Setting key d3b0ee9694a4218a51bd1f5748fbec83
I20260926 13:49:03.039948 1823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46c77a9331e04b998eaf846d3b6c62e8: Bootstrap starting.
I20260926 13:49:03.042120 1823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46c77a9331e04b998eaf846d3b6c62e8: Neither blocks nor log segments found. Creating new log.
I20260926 13:49:03.042762 1823 log.cc:824] T 00000000000000000000000000000000 P 46c77a9331e04b998eaf846d3b6c62e8: Log is configured to *not* fsync() on all Append() calls
I20260926 13:49:03.044685 1823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46c77a9331e04b998eaf846d3b6c62e8: No bootstrap required, opened a new log
Sep 26 13:49:03 dist-test-slave-hc1k krb5kdc[1742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 13:49:03.048439 1823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46c77a9331e04b998eaf846d3b6c62e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c77a9331e04b998eaf846d3b6c62e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45477 } }
I20260926 13:49:03.048655 1823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46c77a9331e04b998eaf846d3b6c62e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:49:03.048710 1823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46c77a9331e04b998eaf846d3b6c62e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c77a9331e04b998eaf846d3b6c62e8, State: Initialized, Role: FOLLOWER
I20260926 13:49:03.049136 1823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46c77a9331e04b998eaf846d3b6c62e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c77a9331e04b998eaf846d3b6c62e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45477 } }
I20260926 13:49:03.049289 1823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46c77a9331e04b998eaf846d3b6c62e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:49:03.049357 1823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46c77a9331e04b998eaf846d3b6c62e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:49:03.049453 1823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46c77a9331e04b998eaf846d3b6c62e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:49:03.050434 1823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46c77a9331e04b998eaf846d3b6c62e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c77a9331e04b998eaf846d3b6c62e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45477 } }
I20260926 13:49:03.050784 1823 leader_election.cc:302] T 00000000000000000000000000000000 P 46c77a9331e04b998eaf846d3b6c62e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46c77a9331e04b998eaf846d3b6c62e8; no voters:
I20260926 13:49:03.051067 1823 leader_election.cc:288] T 00000000000000000000000000000000 P 46c77a9331e04b998eaf846d3b6c62e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:49:03.051156 1828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46c77a9331e04b998eaf846d3b6c62e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:49:03.051358 1828 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46c77a9331e04b998eaf846d3b6c62e8 [term 1 LEADER]: Becoming Leader. State: Replica: 46c77a9331e04b998eaf846d3b6c62e8, State: Running, Role: LEADER
I20260926 13:49:03.051712 1828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46c77a9331e04b998eaf846d3b6c62e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c77a9331e04b998eaf846d3b6c62e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45477 } }
I20260926 13:49:03.052248 1823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46c77a9331e04b998eaf846d3b6c62e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:49:03.053479 1829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46c77a9331e04b998eaf846d3b6c62e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46c77a9331e04b998eaf846d3b6c62e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c77a9331e04b998eaf846d3b6c62e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45477 } } }
I20260926 13:49:03.053511 1830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46c77a9331e04b998eaf846d3b6c62e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46c77a9331e04b998eaf846d3b6c62e8. Latest consensus state: current_term: 1 leader_uuid: "46c77a9331e04b998eaf846d3b6c62e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c77a9331e04b998eaf846d3b6c62e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45477 } } }
I20260926 13:49:03.053635 1829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46c77a9331e04b998eaf846d3b6c62e8 [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:03.053686 1830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46c77a9331e04b998eaf846d3b6c62e8 [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:03.053941 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:03.040765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:40161 (local address 127.0.0.1:45477)
0926 13:49:03.041162 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:03.041176 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:03.041209 (+ 33us) server_negotiation.cc:407] Connection header received
0926 13:49:03.041813 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:03.041829 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:03.042137 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:03.042436 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:03.043243 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:03.044614 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:03.045468 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:03.046113 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:03.046281 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:03.048670 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:03.048676 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:03.048691 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:03.048720 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:03.050587 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:03.051098 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:03.051103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:03.051108 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:03.051239 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:03.051623 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:03.051626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:03.051628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:03.052117 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:03.052353 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:03.052616 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 13:49:03.053005 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":125,"threads_started":1}
I20260926 13:49:03.054060 1838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:49:03.056881 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:49:03.057839 1838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:49:03.062773 1838 catalog_manager.cc:1347] Generated new cluster ID: d979148fd0234b5f8c291ecba592a839
I20260926 13:49:03.062839 1838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:49:03.067109 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:03.058945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:44223 (local address 127.0.0.1:45477)
0926 13:49:03.059046 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:03.059050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:03.059381 (+ 331us) server_negotiation.cc:407] Connection header received
0926 13:49:03.059480 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:03.059483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:03.059528 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:03.059604 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:03.060470 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:03.061267 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:03.062423 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:03.062778 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:03.062897 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:03.064316 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:03.064320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:03.064324 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:03.064347 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:03.065604 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:03.066021 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:03.066024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:03.066026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:03.066094 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:03.066372 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:03.066375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:03.066377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:03.066528 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:03.066647 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:03.066894 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 13:49:03.066973 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 13:49:03.092924 1838 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:49:03.093771 1838 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:49:03.105820 1838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46c77a9331e04b998eaf846d3b6c62e8: Generated new TSK 0
I20260926 13:49:03.106446 1838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:49:03.115985 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 1755
W20260926 13:49:03.120517 1740 connection.cc:535] client connection to 127.0.0.1:45477 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45477): Connection reset by peer (error 104)
2026-09-26T13:49:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:49:03 dist-test-slave-hc1k krb5kdc[1854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:49:03 dist-test-slave-hc1k krb5kdc[1854](info): set up 2 sockets
Sep 26 13:49:03 dist-test-slave-hc1k krb5kdc[1854](info): commencing operation
krb5kdc: starting...
W20260926 13:49:04.055562 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:49:04 dist-test-slave-hc1k krb5kdc[1854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:49:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:49:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:49:04.166247 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46479
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:49:04.269167 1867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:49:04.269400 1867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:49:04.269438 1867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:49:04.273339 1867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:49:04.273386 1867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:49:04.273403 1867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:49:04.273417 1867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:49:04.273432 1867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:49:04.278357 1867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46479
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:49:04.279278 1867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:49:04.280212 1867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:49:04.287948 1872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:04.288002 1873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:49:04.288059 1867 server_base.cc:1034] running on GCE node
W20260926 13:49:04.288132 1875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:49:04.288651 1867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:49:04.289646 1867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:49:04.290827 1867 hybrid_clock.cc:648] HybridClock initialized: now 1790430544290807 us; error 30 us; skew 500 ppm
Sep 26 13:49:04 dist-test-slave-hc1k krb5kdc[1854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:49:04.293282 1867 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:49:04.294081 1867 webserver.cc:460] Webserver started at http://127.0.0.1:43687/ using document root <none> and password file <none>
I20260926 13:49:04.294565 1867 fs_manager.cc:359] Metadata directory not provided
I20260926 13:49:04.294620 1867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:49:04.294799 1867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:49:04.296780 1867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "7038870a08b64ae8af063196e539461d"
format_stamp: "Formatted at 2026-09-26 13:49:04 on dist-test-slave-hc1k"
server_key: "0aeb507202b944459dd69169f0bfdadf"
server_key_iv: "0e9bec5d1a1fff2c08103275e7f7baac"
server_key_version: "encryptionkey@0"
I20260926 13:49:04.297281 1867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "7038870a08b64ae8af063196e539461d"
format_stamp: "Formatted at 2026-09-26 13:49:04 on dist-test-slave-hc1k"
server_key: "0aeb507202b944459dd69169f0bfdadf"
server_key_iv: "0e9bec5d1a1fff2c08103275e7f7baac"
server_key_version: "encryptionkey@0"
I20260926 13:49:04.300612 1867 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:49:04.302716 1882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:49:04.303853 1867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:49:04.303943 1867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "7038870a08b64ae8af063196e539461d"
format_stamp: "Formatted at 2026-09-26 13:49:04 on dist-test-slave-hc1k"
server_key: "0aeb507202b944459dd69169f0bfdadf"
server_key_iv: "0e9bec5d1a1fff2c08103275e7f7baac"
server_key_version: "encryptionkey@0"
I20260926 13:49:04.304039 1867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:49:04.319545 1867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:49:04.322062 1867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:49:04.322211 1867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:49:04.328684 1867 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46479
I20260926 13:49:04.328691 1934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46479 every 8 connection(s)
I20260926 13:49:04.329540 1867 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:49:04.330647 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 1867
I20260926 13:49:04.330778 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:49:04.331038 12969 external_mini_cluster.cc:1428] Setting key 20c17a5828936e6fb7fcbb43da95f0f5
I20260926 13:49:04.332340 1935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:49:04.338763 1935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7038870a08b64ae8af063196e539461d: Bootstrap starting.
I20260926 13:49:04.340672 1938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:04.332342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:45303 (local address 127.0.0.1:46479)
0926 13:49:04.332735 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:04.332752 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:04.332793 (+ 41us) server_negotiation.cc:407] Connection header received
0926 13:49:04.333607 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:04.333629 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:04.334030 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:04.334367 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:04.334751 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:04.336205 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:04.337093 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:04.337767 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:04.337921 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:04.338774 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:04.338780 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:04.338795 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:49:04.339238 (+ 443us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:04.339298 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:49:04.339340 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:04.339461 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:04.339639 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 13:49:04.339797 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":106,"threads_started":1}
I20260926 13:49:04.341269 1935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7038870a08b64ae8af063196e539461d: Neither blocks nor log segments found. Creating new log.
I20260926 13:49:04.342002 1935 log.cc:824] T 00000000000000000000000000000000 P 7038870a08b64ae8af063196e539461d: Log is configured to *not* fsync() on all Append() calls
I20260926 13:49:04.343442 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:49:04.343918 1935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7038870a08b64ae8af063196e539461d: No bootstrap required, opened a new log
I20260926 13:49:04.347319 1935 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7038870a08b64ae8af063196e539461d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7038870a08b64ae8af063196e539461d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46479 } }
I20260926 13:49:04.347518 1935 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7038870a08b64ae8af063196e539461d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:49:04.347589 1935 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7038870a08b64ae8af063196e539461d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7038870a08b64ae8af063196e539461d, State: Initialized, Role: FOLLOWER
I20260926 13:49:04.348032 1935 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7038870a08b64ae8af063196e539461d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7038870a08b64ae8af063196e539461d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46479 } }
I20260926 13:49:04.348191 1935 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7038870a08b64ae8af063196e539461d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:49:04.348253 1935 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7038870a08b64ae8af063196e539461d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:49:04.348345 1935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7038870a08b64ae8af063196e539461d [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:49:04.348783 1938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:04.345254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:38967 (local address 127.0.0.1:46479)
0926 13:49:04.345345 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:04.345349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:04.345541 (+ 192us) server_negotiation.cc:407] Connection header received
0926 13:49:04.345650 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:04.345653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:04.345689 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:04.345745 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:04.346004 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:04.346485 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:04.347261 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:04.347474 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:04.347555 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:04.348385 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:04.348389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:04.348391 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:49:04.348404 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:04.348427 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:49:04.348440 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:04.348516 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:04.348633 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 13:49:04.348663 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 13:49:04.349241 1935 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7038870a08b64ae8af063196e539461d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7038870a08b64ae8af063196e539461d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46479 } }
I20260926 13:49:04.349545 1935 leader_election.cc:302] T 00000000000000000000000000000000 P 7038870a08b64ae8af063196e539461d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7038870a08b64ae8af063196e539461d; no voters:
I20260926 13:49:04.349838 1935 leader_election.cc:288] T 00000000000000000000000000000000 P 7038870a08b64ae8af063196e539461d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:49:04.349936 1945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7038870a08b64ae8af063196e539461d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:49:04.350128 1945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7038870a08b64ae8af063196e539461d [term 1 LEADER]: Becoming Leader. State: Replica: 7038870a08b64ae8af063196e539461d, State: Running, Role: LEADER
I20260926 13:49:04.350463 1945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7038870a08b64ae8af063196e539461d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7038870a08b64ae8af063196e539461d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46479 } }
I20260926 13:49:04.350884 1935 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7038870a08b64ae8af063196e539461d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:49:04.351979 1946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7038870a08b64ae8af063196e539461d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7038870a08b64ae8af063196e539461d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7038870a08b64ae8af063196e539461d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46479 } } }
I20260926 13:49:04.351995 1947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7038870a08b64ae8af063196e539461d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7038870a08b64ae8af063196e539461d. Latest consensus state: current_term: 1 leader_uuid: "7038870a08b64ae8af063196e539461d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7038870a08b64ae8af063196e539461d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46479 } } }
I20260926 13:49:04.352145 1946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7038870a08b64ae8af063196e539461d [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:04.352197 1947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7038870a08b64ae8af063196e539461d [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:04.352540 1956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:49:04.355104 1956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:49:04.359916 1956 catalog_manager.cc:1347] Generated new cluster ID: 7483bdfb928e4ec6a288e8fee62df663
I20260926 13:49:04.359972 1956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:49:04.368764 1956 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:49:04.369899 1956 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:49:04.383692 1956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7038870a08b64ae8af063196e539461d: Generated new TSK 0
I20260926 13:49:04.384464 1956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:49:04.393138 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 1867
W20260926 13:49:04.397651 1852 connection.cc:535] client connection to 127.0.0.1:46479 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46479): Connection reset by peer (error 104)
2026-09-26T13:49:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1277 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:49:04 dist-test-slave-hc1k krb5kdc[1966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:49:04 dist-test-slave-hc1k krb5kdc[1966](info): set up 2 sockets
Sep 26 13:49:04 dist-test-slave-hc1k krb5kdc[1966](info): commencing operation
krb5kdc: starting...
W20260926 13:49:05.339634 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:49:05 dist-test-slave-hc1k krb5kdc[1966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:49:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:49:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:49:05.448503 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36483
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:49:05.552515 1979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:49:05.552749 1979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:49:05.552802 1979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:49:05.556711 1979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:49:05.556761 1979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:49:05.556787 1979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:49:05.556809 1979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:49:05.556836 1979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:49:05.561935 1979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36483
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:49:05.562836 1979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:49:05.563773 1979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:49:05.571398 1979 server_base.cc:1034] running on GCE node
W20260926 13:49:05.571449 1985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:05.571336 1987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:05.571349 1984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:49:05.572081 1979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:49:05.573053 1979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:49:05.574210 1979 hybrid_clock.cc:648] HybridClock initialized: now 1790430545574185 us; error 35 us; skew 500 ppm
Sep 26 13:49:05 dist-test-slave-hc1k krb5kdc[1966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:49:05.576630 1979 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:49:05.577497 1979 webserver.cc:460] Webserver started at http://127.0.0.1:42395/ using document root <none> and password file <none>
I20260926 13:49:05.577982 1979 fs_manager.cc:359] Metadata directory not provided
I20260926 13:49:05.578043 1979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:49:05.578235 1979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:49:05.580186 1979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "b18b3b6007ea4a6fb8101ae557453fe3"
format_stamp: "Formatted at 2026-09-26 13:49:05 on dist-test-slave-hc1k"
server_key: "c3a978978271c858559e428643ffec1a"
server_key_iv: "3dfc5983b8aa3dffde50be580e97697d"
server_key_version: "encryptionkey@0"
I20260926 13:49:05.580698 1979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "b18b3b6007ea4a6fb8101ae557453fe3"
format_stamp: "Formatted at 2026-09-26 13:49:05 on dist-test-slave-hc1k"
server_key: "c3a978978271c858559e428643ffec1a"
server_key_iv: "3dfc5983b8aa3dffde50be580e97697d"
server_key_version: "encryptionkey@0"
I20260926 13:49:05.584061 1979 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:49:05.586273 1994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:49:05.587486 1979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:49:05.587610 1979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "b18b3b6007ea4a6fb8101ae557453fe3"
format_stamp: "Formatted at 2026-09-26 13:49:05 on dist-test-slave-hc1k"
server_key: "c3a978978271c858559e428643ffec1a"
server_key_iv: "3dfc5983b8aa3dffde50be580e97697d"
server_key_version: "encryptionkey@0"
I20260926 13:49:05.587738 1979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:49:05.591933 1979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:49:05.594489 1979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:49:05.594640 1979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:49:05.601095 1979 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36483
I20260926 13:49:05.601110 2046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36483 every 8 connection(s)
I20260926 13:49:05.601925 1979 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:49:05.602936 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 1979
I20260926 13:49:05.603056 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:49:05.603319 12969 external_mini_cluster.cc:1428] Setting key e98352bda85be2727fb468ac69d5c630
I20260926 13:49:05.604739 2047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:49:05.608470 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:05.604582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:47023 (local address 127.0.0.1:36483)
0926 13:49:05.604980 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:05.605010 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:05.605049 (+ 39us) server_negotiation.cc:407] Connection header received
0926 13:49:05.605831 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:05.605853 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:05.606198 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:05.606531 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:05.607142 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:05.607147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:05.607161 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:49:05.607170 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:05.607223 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:49:05.607245 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:05.607378 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:05.607443 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 13:49:05.607586 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":113,"threads_started":1}
I20260926 13:49:05.611105 2047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b18b3b6007ea4a6fb8101ae557453fe3: Bootstrap starting.
I20260926 13:49:05.611162 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:49:05.613579 2047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b18b3b6007ea4a6fb8101ae557453fe3: Neither blocks nor log segments found. Creating new log.
I20260926 13:49:05.614209 2047 log.cc:824] T 00000000000000000000000000000000 P b18b3b6007ea4a6fb8101ae557453fe3: Log is configured to *not* fsync() on all Append() calls
I20260926 13:49:05.614913 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:05.613397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.220:36801 (local address 127.0.0.1:36483)
0926 13:49:05.613506 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:05.613510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:05.613691 (+ 181us) server_negotiation.cc:407] Connection header received
0926 13:49:05.613826 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:05.613828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:05.613857 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:05.613916 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:05.614579 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:05.614582 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:05.614585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:49:05.614596 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:05.614621 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:49:05.614631 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:05.614699 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:05.614773 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 13:49:05.614810 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:49:05.616067 2047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b18b3b6007ea4a6fb8101ae557453fe3: No bootstrap required, opened a new log
I20260926 13:49:05.619503 2047 raft_consensus.cc:348] T 00000000000000000000000000000000 P b18b3b6007ea4a6fb8101ae557453fe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18b3b6007ea4a6fb8101ae557453fe3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36483 } }
I20260926 13:49:05.619737 2047 raft_consensus.cc:374] T 00000000000000000000000000000000 P b18b3b6007ea4a6fb8101ae557453fe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:49:05.619793 2047 raft_consensus.cc:729] T 00000000000000000000000000000000 P b18b3b6007ea4a6fb8101ae557453fe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b18b3b6007ea4a6fb8101ae557453fe3, State: Initialized, Role: FOLLOWER
I20260926 13:49:05.620164 2047 consensus_queue.cc:260] T 00000000000000000000000000000000 P b18b3b6007ea4a6fb8101ae557453fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18b3b6007ea4a6fb8101ae557453fe3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36483 } }
I20260926 13:49:05.620285 2047 raft_consensus.cc:388] T 00000000000000000000000000000000 P b18b3b6007ea4a6fb8101ae557453fe3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:49:05.620343 2047 raft_consensus.cc:482] T 00000000000000000000000000000000 P b18b3b6007ea4a6fb8101ae557453fe3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:49:05.620448 2047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b18b3b6007ea4a6fb8101ae557453fe3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:49:05.621191 2047 raft_consensus.cc:504] T 00000000000000000000000000000000 P b18b3b6007ea4a6fb8101ae557453fe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18b3b6007ea4a6fb8101ae557453fe3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36483 } }
I20260926 13:49:05.621493 2047 leader_election.cc:302] T 00000000000000000000000000000000 P b18b3b6007ea4a6fb8101ae557453fe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b18b3b6007ea4a6fb8101ae557453fe3; no voters:
I20260926 13:49:05.621743 2047 leader_election.cc:288] T 00000000000000000000000000000000 P b18b3b6007ea4a6fb8101ae557453fe3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:49:05.621845 2057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b18b3b6007ea4a6fb8101ae557453fe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:49:05.622030 2057 raft_consensus.cc:686] T 00000000000000000000000000000000 P b18b3b6007ea4a6fb8101ae557453fe3 [term 1 LEADER]: Becoming Leader. State: Replica: b18b3b6007ea4a6fb8101ae557453fe3, State: Running, Role: LEADER
I20260926 13:49:05.622316 2057 consensus_queue.cc:237] T 00000000000000000000000000000000 P b18b3b6007ea4a6fb8101ae557453fe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18b3b6007ea4a6fb8101ae557453fe3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36483 } }
I20260926 13:49:05.622759 2047 sys_catalog.cc:564] T 00000000000000000000000000000000 P b18b3b6007ea4a6fb8101ae557453fe3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:49:05.623824 2059 sys_catalog.cc:455] T 00000000000000000000000000000000 P b18b3b6007ea4a6fb8101ae557453fe3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b18b3b6007ea4a6fb8101ae557453fe3. Latest consensus state: current_term: 1 leader_uuid: "b18b3b6007ea4a6fb8101ae557453fe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18b3b6007ea4a6fb8101ae557453fe3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36483 } } }
I20260926 13:49:05.623942 2059 sys_catalog.cc:458] T 00000000000000000000000000000000 P b18b3b6007ea4a6fb8101ae557453fe3 [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:05.624207 2058 sys_catalog.cc:455] T 00000000000000000000000000000000 P b18b3b6007ea4a6fb8101ae557453fe3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b18b3b6007ea4a6fb8101ae557453fe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18b3b6007ea4a6fb8101ae557453fe3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36483 } } }
I20260926 13:49:05.624291 2058 sys_catalog.cc:458] T 00000000000000000000000000000000 P b18b3b6007ea4a6fb8101ae557453fe3 [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:05.624318 2066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:49:05.627053 2066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:49:05.631757 2066 catalog_manager.cc:1347] Generated new cluster ID: 0ec7a90d485346368aa90c995540008e
I20260926 13:49:05.631815 2066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:49:05.652148 2066 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:49:05.652566 2066 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:49:05.663857 2066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b18b3b6007ea4a6fb8101ae557453fe3: Generated new TSK 0
I20260926 13:49:05.664433 2066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:49:05.672693 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 1979
2026-09-26T13:49:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1279 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:49:05 dist-test-slave-hc1k krb5kdc[2078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:49:05 dist-test-slave-hc1k krb5kdc[2078](info): set up 2 sockets
Sep 26 13:49:05 dist-test-slave-hc1k krb5kdc[2078](info): commencing operation
krb5kdc: starting...
W20260926 13:49:06.619467 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:49:06 dist-test-slave-hc1k krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:49:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:49:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:49:06.726920 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:39523
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:40929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:39523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:49:06.827440 2091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:49:06.827677 2091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:49:06.827725 2091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:49:06.831557 2091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:49:06.831609 2091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:49:06.831633 2091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:49:06.831674 2091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:49:06.831703 2091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:49:06.837767 2091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:40929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:39523
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:39523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:49:06.838655 2091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:49:06.839568 2091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:49:06.847314 2096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:06.847263 2099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:49:06.847394 2091 server_base.cc:1034] running on GCE node
W20260926 13:49:06.847263 2097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:49:06.848066 2091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:49:06.849049 2091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:49:06.850211 2091 hybrid_clock.cc:648] HybridClock initialized: now 1790430546850187 us; error 34 us; skew 500 ppm
Sep 26 13:49:06 dist-test-slave-hc1k krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430546, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:49:06.855837 2091 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:49:06.856606 2091 webserver.cc:460] Webserver started at http://127.12.170.126:35667/ using document root <none> and password file <none>
I20260926 13:49:06.857092 2091 fs_manager.cc:359] Metadata directory not provided
I20260926 13:49:06.857152 2091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:49:06.857322 2091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:49:06.859176 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "fffbb5472b7247129fa024634d4ed07d"
format_stamp: "Formatted at 2026-09-26 13:49:06 on dist-test-slave-hc1k"
server_key: "58fa781714108766930943a9b5f7b56e"
server_key_iv: "7e847a22847e80ff8ffa9d8cf07bdbc0"
server_key_version: "encryptionkey@0"
I20260926 13:49:06.859710 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "fffbb5472b7247129fa024634d4ed07d"
format_stamp: "Formatted at 2026-09-26 13:49:06 on dist-test-slave-hc1k"
server_key: "58fa781714108766930943a9b5f7b56e"
server_key_iv: "7e847a22847e80ff8ffa9d8cf07bdbc0"
server_key_version: "encryptionkey@0"
I20260926 13:49:06.863088 2091 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260926 13:49:06.865299 2106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:49:06.866376 2091 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:49:06.866482 2091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "fffbb5472b7247129fa024634d4ed07d"
format_stamp: "Formatted at 2026-09-26 13:49:06 on dist-test-slave-hc1k"
server_key: "58fa781714108766930943a9b5f7b56e"
server_key_iv: "7e847a22847e80ff8ffa9d8cf07bdbc0"
server_key_version: "encryptionkey@0"
I20260926 13:49:06.866566 2091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:49:06.886119 2091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:49:06.888727 2091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:49:06.888883 2091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:49:06.895040 2091 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:39523
I20260926 13:49:06.895078 2158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:39523 every 8 connection(s)
I20260926 13:49:06.895927 2091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:49:06.898471 2159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:49:06.901660 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 2091
I20260926 13:49:06.901775 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:49:06.902027 12969 external_mini_cluster.cc:1428] Setting key 72d0523d3e3aad4cb92369839fdd9f44
I20260926 13:49:06.903867 2159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffbb5472b7247129fa024634d4ed07d: Bootstrap starting.
I20260926 13:49:06.906445 2159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fffbb5472b7247129fa024634d4ed07d: Neither blocks nor log segments found. Creating new log.
I20260926 13:49:06.907121 2159 log.cc:824] T 00000000000000000000000000000000 P fffbb5472b7247129fa024634d4ed07d: Log is configured to *not* fsync() on all Append() calls
I20260926 13:49:06.908809 2159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffbb5472b7247129fa024634d4ed07d: No bootstrap required, opened a new log
Sep 26 13:49:06 dist-test-slave-hc1k krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:49:06.912490 2159 raft_consensus.cc:348] T 00000000000000000000000000000000 P fffbb5472b7247129fa024634d4ed07d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffbb5472b7247129fa024634d4ed07d" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 39523 } }
I20260926 13:49:06.912691 2159 raft_consensus.cc:374] T 00000000000000000000000000000000 P fffbb5472b7247129fa024634d4ed07d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:49:06.912724 2159 raft_consensus.cc:729] T 00000000000000000000000000000000 P fffbb5472b7247129fa024634d4ed07d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fffbb5472b7247129fa024634d4ed07d, State: Initialized, Role: FOLLOWER
I20260926 13:49:06.913132 2159 consensus_queue.cc:260] T 00000000000000000000000000000000 P fffbb5472b7247129fa024634d4ed07d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffbb5472b7247129fa024634d4ed07d" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 39523 } }
I20260926 13:49:06.913251 2159 raft_consensus.cc:388] T 00000000000000000000000000000000 P fffbb5472b7247129fa024634d4ed07d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:49:06.913352 2159 raft_consensus.cc:482] T 00000000000000000000000000000000 P fffbb5472b7247129fa024634d4ed07d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:49:06.913460 2159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fffbb5472b7247129fa024634d4ed07d [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:49:06.914386 2159 raft_consensus.cc:504] T 00000000000000000000000000000000 P fffbb5472b7247129fa024634d4ed07d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffbb5472b7247129fa024634d4ed07d" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 39523 } }
I20260926 13:49:06.914713 2159 leader_election.cc:302] T 00000000000000000000000000000000 P fffbb5472b7247129fa024634d4ed07d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fffbb5472b7247129fa024634d4ed07d; no voters:
I20260926 13:49:06.915086 2159 leader_election.cc:288] T 00000000000000000000000000000000 P fffbb5472b7247129fa024634d4ed07d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:49:06.915294 2164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fffbb5472b7247129fa024634d4ed07d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:49:06.915525 2164 raft_consensus.cc:686] T 00000000000000000000000000000000 P fffbb5472b7247129fa024634d4ed07d [term 1 LEADER]: Becoming Leader. State: Replica: fffbb5472b7247129fa024634d4ed07d, State: Running, Role: LEADER
I20260926 13:49:06.915882 2164 consensus_queue.cc:237] T 00000000000000000000000000000000 P fffbb5472b7247129fa024634d4ed07d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffbb5472b7247129fa024634d4ed07d" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 39523 } }
I20260926 13:49:06.916157 2162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:06.903311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60189 (local address 127.12.170.126:39523)
0926 13:49:06.903824 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:06.903869 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:06.903908 (+ 39us) server_negotiation.cc:407] Connection header received
0926 13:49:06.904734 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:06.904756 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:06.905182 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:06.905535 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:06.906292 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:06.907352 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:06.908256 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:06.908717 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:06.908856 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:06.911502 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:06.911507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:06.911519 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:06.911543 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:06.913172 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:06.913699 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:06.913704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:06.913708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:06.913808 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:06.914098 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:06.914100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:06.914101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:06.914468 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:06.914634 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:06.914978 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 13:49:06.915187 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":112,"threads_started":1}
I20260926 13:49:06.916210 2159 sys_catalog.cc:564] T 00000000000000000000000000000000 P fffbb5472b7247129fa024634d4ed07d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:49:06.917398 2165 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffbb5472b7247129fa024634d4ed07d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fffbb5472b7247129fa024634d4ed07d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffbb5472b7247129fa024634d4ed07d" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 39523 } } }
I20260926 13:49:06.917516 2165 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffbb5472b7247129fa024634d4ed07d [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:06.917804 2166 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffbb5472b7247129fa024634d4ed07d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fffbb5472b7247129fa024634d4ed07d. Latest consensus state: current_term: 1 leader_uuid: "fffbb5472b7247129fa024634d4ed07d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffbb5472b7247129fa024634d4ed07d" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 39523 } } }
I20260926 13:49:06.917888 2166 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffbb5472b7247129fa024634d4ed07d [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:06.919971 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 13:49:06.920969 2179 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fffbb5472b7247129fa024634d4ed07d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:49:06.921030 2179 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:49:06.921135 2175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:49:06.921941 2175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:49:06.927309 2175 catalog_manager.cc:1347] Generated new cluster ID: 3e9bb8f10faf4026ae4517b732a7238a
I20260926 13:49:06.927368 2175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:49:06.929942 2162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:06.921984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41821 (local address 127.12.170.126:39523)
0926 13:49:06.922196 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:06.922200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:06.922250 (+ 50us) server_negotiation.cc:407] Connection header received
0926 13:49:06.922381 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:06.922383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:06.922424 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:06.922501 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:06.923240 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:06.923908 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:06.924654 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:06.924877 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:06.924958 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:06.926532 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:06.926535 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:06.926538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:06.926558 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:06.928345 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:06.928849 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:06.928851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:06.928853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:06.928921 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:06.929202 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:06.929205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:06.929207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:06.929368 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:06.929483 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:06.929718 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 13:49:06.929796 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260926 13:49:06.933243 2175 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:49:06.934028 2175 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:49:06.941434 2175 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fffbb5472b7247129fa024634d4ed07d: Generated new TSK 0
I20260926 13:49:06.942075 2175 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:49:06.949257 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 2091
W20260926 13:49:06.953490 2076 connection.cc:535] client connection to 127.12.170.126:39523 recv error: Network error: failed to read from TLS socket (remote: 127.12.170.126:39523): Connection reset by peer (error 104)
2026-09-26T13:49:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1276 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:49:06 dist-test-slave-hc1k krb5kdc[2190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:49:06 dist-test-slave-hc1k krb5kdc[2190](info): set up 2 sockets
Sep 26 13:49:06 dist-test-slave-hc1k krb5kdc[2190](info): commencing operation
krb5kdc: starting...
W20260926 13:49:07.891553 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:49:07 dist-test-slave-hc1k krb5kdc[2190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:49:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:49:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:49:08.000653 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:45415
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:35321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:45415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:49:08.101624 2203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:49:08.101840 2203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:49:08.101877 2203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:49:08.105804 2203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:49:08.105857 2203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:49:08.105875 2203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:49:08.105890 2203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:49:08.105902 2203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:49:08.110687 2203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:35321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:45415
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:45415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:49:08.111467 2203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:49:08.112434 2203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:49:08.120185 2209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:08.120180 2211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:08.120180 2208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:49:08.120493 2203 server_base.cc:1034] running on GCE node
I20260926 13:49:08.120982 2203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:49:08.121955 2203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:49:08.123135 2203 hybrid_clock.cc:648] HybridClock initialized: now 1790430548123112 us; error 32 us; skew 500 ppm
Sep 26 13:49:08 dist-test-slave-hc1k krb5kdc[2190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430548, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:49:08.125801 2203 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:49:08.126649 2203 webserver.cc:460] Webserver started at http://127.12.170.126:43977/ using document root <none> and password file <none>
I20260926 13:49:08.127173 2203 fs_manager.cc:359] Metadata directory not provided
I20260926 13:49:08.127277 2203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:49:08.127480 2203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:49:08.129415 2203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "16ddd16132cc452e9d135839ffaa5a2f"
format_stamp: "Formatted at 2026-09-26 13:49:08 on dist-test-slave-hc1k"
server_key: "149ec5af2966e5dbbc337a6fb36330ff"
server_key_iv: "951a6994d01abeeca1e60932fbea9bc1"
server_key_version: "encryptionkey@0"
I20260926 13:49:08.129948 2203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "16ddd16132cc452e9d135839ffaa5a2f"
format_stamp: "Formatted at 2026-09-26 13:49:08 on dist-test-slave-hc1k"
server_key: "149ec5af2966e5dbbc337a6fb36330ff"
server_key_iv: "951a6994d01abeeca1e60932fbea9bc1"
server_key_version: "encryptionkey@0"
I20260926 13:49:08.133469 2203 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:49:08.135730 2218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:49:08.136837 2203 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 13:49:08.136947 2203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "16ddd16132cc452e9d135839ffaa5a2f"
format_stamp: "Formatted at 2026-09-26 13:49:08 on dist-test-slave-hc1k"
server_key: "149ec5af2966e5dbbc337a6fb36330ff"
server_key_iv: "951a6994d01abeeca1e60932fbea9bc1"
server_key_version: "encryptionkey@0"
I20260926 13:49:08.137030 2203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:49:08.163256 2203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:49:08.166038 2203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:49:08.166185 2203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:49:08.172341 2203 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:45415
I20260926 13:49:08.172366 2270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:45415 every 8 connection(s)
I20260926 13:49:08.173147 2203 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:49:08.175408 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 2203
I20260926 13:49:08.175544 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:49:08.175593 2271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:49:08.175890 12969 external_mini_cluster.cc:1428] Setting key 3eb4ef85034ccff19619504599491ad5
I20260926 13:49:08.182479 2271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16ddd16132cc452e9d135839ffaa5a2f: Bootstrap starting.
I20260926 13:49:08.184747 2271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16ddd16132cc452e9d135839ffaa5a2f: Neither blocks nor log segments found. Creating new log.
I20260926 13:49:08.185343 2274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:08.177078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59485 (local address 127.12.170.126:45415)
0926 13:49:08.177478 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:08.177496 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:08.177540 (+ 44us) server_negotiation.cc:407] Connection header received
0926 13:49:08.178386 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:08.178408 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:08.178833 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:08.179193 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:08.179593 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:08.181091 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:08.182273 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:08.182866 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:08.182982 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:08.183827 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:08.183831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:08.183842 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:49:08.184181 (+ 339us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:08.184229 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:49:08.184250 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:08.184378 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:08.184515 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 13:49:08.184633 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":118,"threads_started":1}
I20260926 13:49:08.185418 2271 log.cc:824] T 00000000000000000000000000000000 P 16ddd16132cc452e9d135839ffaa5a2f: Log is configured to *not* fsync() on all Append() calls
I20260926 13:49:08.187325 2271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16ddd16132cc452e9d135839ffaa5a2f: No bootstrap required, opened a new log
I20260926 13:49:08.188091 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:49:08.191604 2271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16ddd16132cc452e9d135839ffaa5a2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ddd16132cc452e9d135839ffaa5a2f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45415 } }
I20260926 13:49:08.191857 2271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16ddd16132cc452e9d135839ffaa5a2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:49:08.191924 2271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16ddd16132cc452e9d135839ffaa5a2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ddd16132cc452e9d135839ffaa5a2f, State: Initialized, Role: FOLLOWER
I20260926 13:49:08.192327 2271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16ddd16132cc452e9d135839ffaa5a2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ddd16132cc452e9d135839ffaa5a2f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45415 } }
I20260926 13:49:08.192477 2271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16ddd16132cc452e9d135839ffaa5a2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:49:08.192579 2271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16ddd16132cc452e9d135839ffaa5a2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:49:08.192679 2271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16ddd16132cc452e9d135839ffaa5a2f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:49:08.193771 2271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16ddd16132cc452e9d135839ffaa5a2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ddd16132cc452e9d135839ffaa5a2f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45415 } }
I20260926 13:49:08.194118 2271 leader_election.cc:302] T 00000000000000000000000000000000 P 16ddd16132cc452e9d135839ffaa5a2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16ddd16132cc452e9d135839ffaa5a2f; no voters:
I20260926 13:49:08.194339 2274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:08.190193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59089 (local address 127.12.170.126:45415)
0926 13:49:08.190298 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:08.190302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:08.190488 (+ 186us) server_negotiation.cc:407] Connection header received
0926 13:49:08.190561 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:08.190564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:08.190603 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:08.190680 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:08.190865 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:08.191565 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:08.192518 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:08.192844 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:08.192913 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:08.193815 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:08.193832 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:08.193840 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:49:08.193925 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:08.193965 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:49:08.193979 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:08.194053 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:08.194186 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 13:49:08.194217 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 13:49:08.194578 2271 leader_election.cc:288] T 00000000000000000000000000000000 P 16ddd16132cc452e9d135839ffaa5a2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:49:08.194900 2282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16ddd16132cc452e9d135839ffaa5a2f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:49:08.195073 2282 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16ddd16132cc452e9d135839ffaa5a2f [term 1 LEADER]: Becoming Leader. State: Replica: 16ddd16132cc452e9d135839ffaa5a2f, State: Running, Role: LEADER
I20260926 13:49:08.195421 2282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16ddd16132cc452e9d135839ffaa5a2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ddd16132cc452e9d135839ffaa5a2f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45415 } }
I20260926 13:49:08.195824 2271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16ddd16132cc452e9d135839ffaa5a2f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:49:08.197422 2282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16ddd16132cc452e9d135839ffaa5a2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16ddd16132cc452e9d135839ffaa5a2f. Latest consensus state: current_term: 1 leader_uuid: "16ddd16132cc452e9d135839ffaa5a2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ddd16132cc452e9d135839ffaa5a2f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45415 } } }
I20260926 13:49:08.197574 2282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16ddd16132cc452e9d135839ffaa5a2f [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:08.198021 2292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:49:08.198092 2283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16ddd16132cc452e9d135839ffaa5a2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16ddd16132cc452e9d135839ffaa5a2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ddd16132cc452e9d135839ffaa5a2f" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45415 } } }
I20260926 13:49:08.198233 2283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16ddd16132cc452e9d135839ffaa5a2f [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:08.200904 2292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:49:08.206121 2292 catalog_manager.cc:1347] Generated new cluster ID: 4b1092194d9740c29901c8e56d2f78ca
I20260926 13:49:08.206171 2292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:49:08.213042 2292 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:49:08.213968 2292 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:49:08.232821 2292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16ddd16132cc452e9d135839ffaa5a2f: Generated new TSK 0
I20260926 13:49:08.233435 2292 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:49:08.243189 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 2203
W20260926 13:49:08.247624 2188 connection.cc:535] client connection to 127.12.170.126:45415 recv error: Network error: failed to read from TLS socket (remote: 127.12.170.126:45415): Connection reset by peer (error 104)
2026-09-26T13:49:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:49:08 dist-test-slave-hc1k krb5kdc[2302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:49:08 dist-test-slave-hc1k krb5kdc[2302](info): set up 2 sockets
Sep 26 13:49:08 dist-test-slave-hc1k krb5kdc[2302](info): commencing operation
krb5kdc: starting...
W20260926 13:49:09.183805 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:49:09 dist-test-slave-hc1k krb5kdc[2302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:49:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:49:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:49:09.291724 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:42157
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:35731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:42157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:49:09.393160 2315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:49:09.393394 2315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:49:09.393436 2315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:49:09.397373 2315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:49:09.397424 2315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:49:09.397441 2315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:49:09.397459 2315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:49:09.397486 2315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:49:09.402458 2315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:35731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:42157
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:42157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:49:09.403335 2315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:49:09.404230 2315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:49:09.412112 2323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:09.412124 2320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:09.412148 2321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:49:09.412217 2315 server_base.cc:1034] running on GCE node
I20260926 13:49:09.412887 2315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:49:09.413847 2315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:49:09.415002 2315 hybrid_clock.cc:648] HybridClock initialized: now 1790430549414991 us; error 26 us; skew 500 ppm
Sep 26 13:49:09 dist-test-slave-hc1k krb5kdc[2302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430549, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:49:09.417519 2315 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:49:09.418296 2315 webserver.cc:460] Webserver started at http://127.12.170.126:36745/ using document root <none> and password file <none>
I20260926 13:49:09.418768 2315 fs_manager.cc:359] Metadata directory not provided
I20260926 13:49:09.418823 2315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:49:09.418999 2315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:49:09.420866 2315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "51a691d1b5ae4081a0b2b448271df756"
format_stamp: "Formatted at 2026-09-26 13:49:09 on dist-test-slave-hc1k"
server_key: "c5f36adb3253fd2f76e8f9428ad4ec25"
server_key_iv: "b3667972266a6244973cb044bbe8e311"
server_key_version: "encryptionkey@0"
I20260926 13:49:09.421355 2315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "51a691d1b5ae4081a0b2b448271df756"
format_stamp: "Formatted at 2026-09-26 13:49:09 on dist-test-slave-hc1k"
server_key: "c5f36adb3253fd2f76e8f9428ad4ec25"
server_key_iv: "b3667972266a6244973cb044bbe8e311"
server_key_version: "encryptionkey@0"
I20260926 13:49:09.424615 2315 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:49:09.426826 2330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:49:09.428189 2315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:49:09.428299 2315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "51a691d1b5ae4081a0b2b448271df756"
format_stamp: "Formatted at 2026-09-26 13:49:09 on dist-test-slave-hc1k"
server_key: "c5f36adb3253fd2f76e8f9428ad4ec25"
server_key_iv: "b3667972266a6244973cb044bbe8e311"
server_key_version: "encryptionkey@0"
I20260926 13:49:09.428383 2315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:49:09.432524 2315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:49:09.435410 2315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:49:09.435551 2315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:49:09.441962 2315 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:42157
I20260926 13:49:09.441963 2382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:42157 every 8 connection(s)
I20260926 13:49:09.442819 2315 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:49:09.445363 2383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:49:09.446101 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 2315
I20260926 13:49:09.446211 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:49:09.446471 12969 external_mini_cluster.cc:1428] Setting key efd940f11879d7055cc2d368a0fec60f
I20260926 13:49:09.451417 2383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51a691d1b5ae4081a0b2b448271df756: Bootstrap starting.
I20260926 13:49:09.451623 2386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:09.447724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45721 (local address 127.12.170.126:42157)
0926 13:49:09.448111 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:09.448129 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:09.448169 (+ 40us) server_negotiation.cc:407] Connection header received
0926 13:49:09.448951 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:09.448974 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:09.449316 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:09.449658 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:09.450239 (+ 581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:09.450245 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:09.450272 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:49:09.450283 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:09.450342 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:49:09.450367 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:09.450501 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:09.450631 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 13:49:09.450766 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":99,"threads_started":1}
I20260926 13:49:09.453680 2383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51a691d1b5ae4081a0b2b448271df756: Neither blocks nor log segments found. Creating new log.
I20260926 13:49:09.454246 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:49:09.454437 2383 log.cc:824] T 00000000000000000000000000000000 P 51a691d1b5ae4081a0b2b448271df756: Log is configured to *not* fsync() on all Append() calls
I20260926 13:49:09.456408 2383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51a691d1b5ae4081a0b2b448271df756: No bootstrap required, opened a new log
I20260926 13:49:09.457727 2386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:09.456287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34485 (local address 127.12.170.126:42157)
0926 13:49:09.456417 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:09.456421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:09.456629 (+ 208us) server_negotiation.cc:407] Connection header received
0926 13:49:09.456730 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:09.456733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:09.456765 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:09.456836 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:09.457401 (+ 565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:09.457405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:09.457407 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:49:09.457418 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:09.457442 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:49:09.457454 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:09.457546 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:09.457590 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 13:49:09.457621 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 13:49:09.459992 2383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51a691d1b5ae4081a0b2b448271df756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51a691d1b5ae4081a0b2b448271df756" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 42157 } }
I20260926 13:49:09.460183 2383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51a691d1b5ae4081a0b2b448271df756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:49:09.460219 2383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51a691d1b5ae4081a0b2b448271df756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a691d1b5ae4081a0b2b448271df756, State: Initialized, Role: FOLLOWER
I20260926 13:49:09.460542 2383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51a691d1b5ae4081a0b2b448271df756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51a691d1b5ae4081a0b2b448271df756" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 42157 } }
I20260926 13:49:09.460649 2383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51a691d1b5ae4081a0b2b448271df756 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:49:09.460682 2383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51a691d1b5ae4081a0b2b448271df756 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:49:09.460745 2383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51a691d1b5ae4081a0b2b448271df756 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:49:09.461449 2383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51a691d1b5ae4081a0b2b448271df756 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51a691d1b5ae4081a0b2b448271df756" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 42157 } }
I20260926 13:49:09.461709 2383 leader_election.cc:302] T 00000000000000000000000000000000 P 51a691d1b5ae4081a0b2b448271df756 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51a691d1b5ae4081a0b2b448271df756; no voters:
I20260926 13:49:09.461961 2383 leader_election.cc:288] T 00000000000000000000000000000000 P 51a691d1b5ae4081a0b2b448271df756 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:49:09.462056 2393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51a691d1b5ae4081a0b2b448271df756 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:49:09.462293 2393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51a691d1b5ae4081a0b2b448271df756 [term 1 LEADER]: Becoming Leader. State: Replica: 51a691d1b5ae4081a0b2b448271df756, State: Running, Role: LEADER
I20260926 13:49:09.462616 2393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51a691d1b5ae4081a0b2b448271df756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51a691d1b5ae4081a0b2b448271df756" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 42157 } }
I20260926 13:49:09.463042 2383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51a691d1b5ae4081a0b2b448271df756 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:49:09.464126 2394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51a691d1b5ae4081a0b2b448271df756 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51a691d1b5ae4081a0b2b448271df756" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51a691d1b5ae4081a0b2b448271df756" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 42157 } } }
I20260926 13:49:09.464382 2395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51a691d1b5ae4081a0b2b448271df756 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51a691d1b5ae4081a0b2b448271df756. Latest consensus state: current_term: 1 leader_uuid: "51a691d1b5ae4081a0b2b448271df756" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51a691d1b5ae4081a0b2b448271df756" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 42157 } } }
I20260926 13:49:09.464495 2394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51a691d1b5ae4081a0b2b448271df756 [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:09.464531 2395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51a691d1b5ae4081a0b2b448271df756 [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:09.465312 2407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:49:09.467777 2407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:49:09.473129 2407 catalog_manager.cc:1347] Generated new cluster ID: e6e3b8470ff345f7985eae8a71fd447b
I20260926 13:49:09.473178 2407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:49:09.482869 2407 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:49:09.483181 2407 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:49:09.490693 2407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51a691d1b5ae4081a0b2b448271df756: Generated new TSK 0
I20260926 13:49:09.491425 2407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:49:09.502936 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 2315
2026-09-26T13:49:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1259 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:49:09 dist-test-slave-hc1k krb5kdc[2414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:49:09 dist-test-slave-hc1k krb5kdc[2414](info): set up 2 sockets
Sep 26 13:49:09 dist-test-slave-hc1k krb5kdc[2414](info): commencing operation
krb5kdc: starting...
W20260926 13:49:10.451522 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:49:10 dist-test-slave-hc1k krb5kdc[2414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:49:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:49:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:49:10.558405 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:45325
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:41195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:45325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:49:10.662137 2427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:49:10.662360 2427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:49:10.662397 2427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:49:10.666224 2427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:49:10.666277 2427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:49:10.666296 2427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:49:10.666309 2427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:49:10.666322 2427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:49:10.671169 2427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:41195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:45325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:45325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:49:10.672031 2427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:49:10.672856 2427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:49:10.680226 2435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:10.680243 2433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:10.680254 2432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:49:10.680408 2427 server_base.cc:1034] running on GCE node
I20260926 13:49:10.681013 2427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:49:10.681986 2427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:49:10.683147 2427 hybrid_clock.cc:648] HybridClock initialized: now 1790430550683129 us; error 28 us; skew 500 ppm
Sep 26 13:49:10 dist-test-slave-hc1k krb5kdc[2414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430550, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:49:10.685602 2427 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:49:10.686344 2427 webserver.cc:460] Webserver started at http://127.12.170.126:38085/ using document root <none> and password file <none>
I20260926 13:49:10.686801 2427 fs_manager.cc:359] Metadata directory not provided
I20260926 13:49:10.686862 2427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:49:10.687038 2427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:49:10.688881 2427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "1f7c6b67b01a49f4a3a58740cbf03dd6"
format_stamp: "Formatted at 2026-09-26 13:49:10 on dist-test-slave-hc1k"
server_key: "2e6e08c9766520b05aef0dae9aa5e91e"
server_key_iv: "d87f0b1d4456049ebae68a5635c813b4"
server_key_version: "encryptionkey@0"
I20260926 13:49:10.689363 2427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "1f7c6b67b01a49f4a3a58740cbf03dd6"
format_stamp: "Formatted at 2026-09-26 13:49:10 on dist-test-slave-hc1k"
server_key: "2e6e08c9766520b05aef0dae9aa5e91e"
server_key_iv: "d87f0b1d4456049ebae68a5635c813b4"
server_key_version: "encryptionkey@0"
I20260926 13:49:10.692591 2427 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:49:10.694752 2442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:49:10.695839 2427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:49:10.695940 2427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "1f7c6b67b01a49f4a3a58740cbf03dd6"
format_stamp: "Formatted at 2026-09-26 13:49:10 on dist-test-slave-hc1k"
server_key: "2e6e08c9766520b05aef0dae9aa5e91e"
server_key_iv: "d87f0b1d4456049ebae68a5635c813b4"
server_key_version: "encryptionkey@0"
I20260926 13:49:10.696018 2427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:49:10.723762 2427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:49:10.726439 2427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:49:10.726591 2427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:49:10.732645 2427 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:45325
I20260926 13:49:10.732687 2494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:45325 every 8 connection(s)
I20260926 13:49:10.733531 2427 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:49:10.736049 2495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:49:10.741127 2495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f7c6b67b01a49f4a3a58740cbf03dd6: Bootstrap starting.
I20260926 13:49:10.743251 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 2427
I20260926 13:49:10.743378 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:49:10.743443 2495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f7c6b67b01a49f4a3a58740cbf03dd6: Neither blocks nor log segments found. Creating new log.
I20260926 13:49:10.743635 12969 external_mini_cluster.cc:1428] Setting key 044422e35c4f0a9a70c52784b08fc334
I20260926 13:49:10.744256 2495 log.cc:824] T 00000000000000000000000000000000 P 1f7c6b67b01a49f4a3a58740cbf03dd6: Log is configured to *not* fsync() on all Append() calls
I20260926 13:49:10.746047 2495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f7c6b67b01a49f4a3a58740cbf03dd6: No bootstrap required, opened a new log
I20260926 13:49:10.749878 2495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f7c6b67b01a49f4a3a58740cbf03dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f7c6b67b01a49f4a3a58740cbf03dd6" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45325 } }
I20260926 13:49:10.750133 2495 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f7c6b67b01a49f4a3a58740cbf03dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:49:10.750192 2495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f7c6b67b01a49f4a3a58740cbf03dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f7c6b67b01a49f4a3a58740cbf03dd6, State: Initialized, Role: FOLLOWER
I20260926 13:49:10.750602 2495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f7c6b67b01a49f4a3a58740cbf03dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f7c6b67b01a49f4a3a58740cbf03dd6" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45325 } }
I20260926 13:49:10.750725 2495 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f7c6b67b01a49f4a3a58740cbf03dd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:49:10.750784 2495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f7c6b67b01a49f4a3a58740cbf03dd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:49:10.750876 2495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f7c6b67b01a49f4a3a58740cbf03dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:49:10.751736 2495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f7c6b67b01a49f4a3a58740cbf03dd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f7c6b67b01a49f4a3a58740cbf03dd6" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45325 } }
Sep 26 13:49:10 dist-test-slave-hc1k krb5kdc[2414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:49:10.752079 2495 leader_election.cc:302] T 00000000000000000000000000000000 P 1f7c6b67b01a49f4a3a58740cbf03dd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f7c6b67b01a49f4a3a58740cbf03dd6; no voters:
I20260926 13:49:10.752377 2495 leader_election.cc:288] T 00000000000000000000000000000000 P 1f7c6b67b01a49f4a3a58740cbf03dd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:49:10.752449 2500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f7c6b67b01a49f4a3a58740cbf03dd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:49:10.752605 2500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f7c6b67b01a49f4a3a58740cbf03dd6 [term 1 LEADER]: Becoming Leader. State: Replica: 1f7c6b67b01a49f4a3a58740cbf03dd6, State: Running, Role: LEADER
I20260926 13:49:10.752952 2500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f7c6b67b01a49f4a3a58740cbf03dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f7c6b67b01a49f4a3a58740cbf03dd6" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45325 } }
I20260926 13:49:10.753563 2495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f7c6b67b01a49f4a3a58740cbf03dd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:49:10.754551 2501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f7c6b67b01a49f4a3a58740cbf03dd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f7c6b67b01a49f4a3a58740cbf03dd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f7c6b67b01a49f4a3a58740cbf03dd6" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45325 } } }
I20260926 13:49:10.754942 2501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f7c6b67b01a49f4a3a58740cbf03dd6 [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:10.754834 2502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f7c6b67b01a49f4a3a58740cbf03dd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f7c6b67b01a49f4a3a58740cbf03dd6. Latest consensus state: current_term: 1 leader_uuid: "1f7c6b67b01a49f4a3a58740cbf03dd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f7c6b67b01a49f4a3a58740cbf03dd6" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 45325 } } }
I20260926 13:49:10.755259 2502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f7c6b67b01a49f4a3a58740cbf03dd6 [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:10.757507 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:10.744704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50207 (local address 127.12.170.126:45325)
0926 13:49:10.745124 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:10.745142 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:10.745199 (+ 57us) server_negotiation.cc:407] Connection header received
0926 13:49:10.745801 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:10.745826 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:10.746157 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:10.746427 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:10.747248 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:10.748222 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:10.749387 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:10.749834 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:10.749939 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:10.752604 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:10.752610 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:10.752627 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:10.752658 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:10.754406 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:10.754918 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:10.754924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:10.754930 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:10.755091 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:10.755429 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:10.755432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:10.755433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:10.756000 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:10.756189 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:10.756428 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 13:49:10.756694 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":132,"threads_started":1}
W20260926 13:49:10.758246 2515 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1f7c6b67b01a49f4a3a58740cbf03dd6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:49:10.758307 2515 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:49:10.758399 2512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:49:10.759224 2512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:49:10.760310 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:49:10.764153 2512 catalog_manager.cc:1347] Generated new cluster ID: 05a32e98da384c6abadf06771f6fff43
I20260926 13:49:10.764201 2512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:49:10.770263 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:10.762356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51607 (local address 127.12.170.126:45325)
0926 13:49:10.762448 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:10.762452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:10.762808 (+ 356us) server_negotiation.cc:407] Connection header received
0926 13:49:10.762929 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:10.762932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:10.762966 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:10.763024 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:10.763633 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:10.764318 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:10.765431 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:10.765790 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:10.765850 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:10.767322 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:10.767325 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:10.767328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:10.767349 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:10.768791 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:10.769185 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:10.769188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:10.769190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:10.769253 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:10.769480 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:10.769482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:10.769484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:10.769655 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:10.769736 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:10.770028 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 13:49:10.770124 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 13:49:10.779547 2512 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:49:10.780388 2512 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:49:10.785202 2512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f7c6b67b01a49f4a3a58740cbf03dd6: Generated new TSK 0
I20260926 13:49:10.785758 2512 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:49:10.791355 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 2427
2026-09-26T13:49:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1288 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:49:10 dist-test-slave-hc1k krb5kdc[2526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:49:10 dist-test-slave-hc1k krb5kdc[2526](info): set up 2 sockets
Sep 26 13:49:10 dist-test-slave-hc1k krb5kdc[2526](info): commencing operation
krb5kdc: starting...
W20260926 13:49:11.739506 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:49:11 dist-test-slave-hc1k krb5kdc[2526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:49:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:49:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:49:11.848554 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:41787
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:36915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:41787
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:49:11.951488 2539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:49:11.951793 2539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:49:11.951875 2539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:49:11.955871 2539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:49:11.955930 2539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:49:11.955962 2539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:49:11.955999 2539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:49:11.956032 2539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:49:11.961067 2539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:36915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:41787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:41787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:49:11.961989 2539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:49:11.962920 2539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:49:11.970682 2544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:11.970721 2547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:11.970693 2545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:49:11.970853 2539 server_base.cc:1034] running on GCE node
I20260926 13:49:11.971541 2539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:49:11.972613 2539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:49:11.973785 2539 hybrid_clock.cc:648] HybridClock initialized: now 1790430551973775 us; error 32 us; skew 500 ppm
Sep 26 13:49:11 dist-test-slave-hc1k krb5kdc[2526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430551, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:49:11.979558 2539 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:49:11.980387 2539 webserver.cc:460] Webserver started at http://127.12.170.126:37223/ using document root <none> and password file <none>
I20260926 13:49:11.980873 2539 fs_manager.cc:359] Metadata directory not provided
I20260926 13:49:11.980934 2539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:49:11.981103 2539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:49:11.982961 2539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "8da52cc844d6430b898a1e5035d40dc9"
format_stamp: "Formatted at 2026-09-26 13:49:11 on dist-test-slave-hc1k"
server_key: "60440d30f4818ce68d36d57cc6a6e64c"
server_key_iv: "bf7eb30b2bfe424abfc80343cb8803f6"
server_key_version: "encryptionkey@0"
I20260926 13:49:11.983454 2539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "8da52cc844d6430b898a1e5035d40dc9"
format_stamp: "Formatted at 2026-09-26 13:49:11 on dist-test-slave-hc1k"
server_key: "60440d30f4818ce68d36d57cc6a6e64c"
server_key_iv: "bf7eb30b2bfe424abfc80343cb8803f6"
server_key_version: "encryptionkey@0"
I20260926 13:49:11.986809 2539 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 13:49:11.989005 2554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:49:11.990147 2539 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:49:11.990252 2539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "8da52cc844d6430b898a1e5035d40dc9"
format_stamp: "Formatted at 2026-09-26 13:49:11 on dist-test-slave-hc1k"
server_key: "60440d30f4818ce68d36d57cc6a6e64c"
server_key_iv: "bf7eb30b2bfe424abfc80343cb8803f6"
server_key_version: "encryptionkey@0"
I20260926 13:49:11.990356 2539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:49:12.021739 2539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:49:12.024309 2539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:49:12.024495 2539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:49:12.030745 2539 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:41787
I20260926 13:49:12.030786 2606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:41787 every 8 connection(s)
I20260926 13:49:12.031611 2539 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:49:12.033404 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 2539
I20260926 13:49:12.033525 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:49:12.033800 12969 external_mini_cluster.cc:1428] Setting key 4a6e271adeaba6cca71cff56ec8ccc66
I20260926 13:49:12.034372 2607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:49:12.040028 2607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8da52cc844d6430b898a1e5035d40dc9: Bootstrap starting.
I20260926 13:49:12.041731 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:12.035005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49315 (local address 127.12.170.126:41787)
0926 13:49:12.035437 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:12.035450 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:12.035482 (+ 32us) server_negotiation.cc:407] Connection header received
0926 13:49:12.036104 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:12.036120 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:12.036442 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:12.036726 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:12.037093 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:12.038025 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:12.038867 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:12.039305 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:12.039447 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:12.040251 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:12.040255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:12.040266 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:49:12.040613 (+ 347us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:12.040664 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:49:12.040685 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:12.040764 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:12.040896 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 13:49:12.041083 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":164,"threads_started":1}
I20260926 13:49:12.042124 2607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8da52cc844d6430b898a1e5035d40dc9: Neither blocks nor log segments found. Creating new log.
I20260926 13:49:12.042840 2607 log.cc:824] T 00000000000000000000000000000000 P 8da52cc844d6430b898a1e5035d40dc9: Log is configured to *not* fsync() on all Append() calls
I20260926 13:49:12.044330 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:49:12.044770 2607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8da52cc844d6430b898a1e5035d40dc9: No bootstrap required, opened a new log
I20260926 13:49:12.048461 2607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8da52cc844d6430b898a1e5035d40dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da52cc844d6430b898a1e5035d40dc9" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 41787 } }
I20260926 13:49:12.048725 2607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8da52cc844d6430b898a1e5035d40dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:49:12.048799 2607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8da52cc844d6430b898a1e5035d40dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da52cc844d6430b898a1e5035d40dc9, State: Initialized, Role: FOLLOWER
I20260926 13:49:12.049206 2607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8da52cc844d6430b898a1e5035d40dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da52cc844d6430b898a1e5035d40dc9" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 41787 } }
I20260926 13:49:12.049372 2607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8da52cc844d6430b898a1e5035d40dc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:49:12.049445 2607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8da52cc844d6430b898a1e5035d40dc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:49:12.049551 2607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8da52cc844d6430b898a1e5035d40dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:49:12.050293 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:12.046355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53989 (local address 127.12.170.126:41787)
0926 13:49:12.046451 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:12.046454 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:12.046567 (+ 113us) server_negotiation.cc:407] Connection header received
0926 13:49:12.046673 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:12.046675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:12.046706 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:12.046761 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:12.047010 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:12.047826 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:12.048903 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:12.049160 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:12.049234 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:12.049963 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:12.049966 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:12.049968 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:49:12.049977 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:12.049998 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:49:12.050007 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:12.050060 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:12.050166 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 13:49:12.050201 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 13:49:12.050619 2607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8da52cc844d6430b898a1e5035d40dc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da52cc844d6430b898a1e5035d40dc9" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 41787 } }
I20260926 13:49:12.050963 2607 leader_election.cc:302] T 00000000000000000000000000000000 P 8da52cc844d6430b898a1e5035d40dc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8da52cc844d6430b898a1e5035d40dc9; no voters:
I20260926 13:49:12.051311 2607 leader_election.cc:288] T 00000000000000000000000000000000 P 8da52cc844d6430b898a1e5035d40dc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:49:12.051393 2617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8da52cc844d6430b898a1e5035d40dc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:49:12.051565 2617 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8da52cc844d6430b898a1e5035d40dc9 [term 1 LEADER]: Becoming Leader. State: Replica: 8da52cc844d6430b898a1e5035d40dc9, State: Running, Role: LEADER
I20260926 13:49:12.051887 2617 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8da52cc844d6430b898a1e5035d40dc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da52cc844d6430b898a1e5035d40dc9" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 41787 } }
I20260926 13:49:12.052543 2607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8da52cc844d6430b898a1e5035d40dc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:49:12.053450 2618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8da52cc844d6430b898a1e5035d40dc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8da52cc844d6430b898a1e5035d40dc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da52cc844d6430b898a1e5035d40dc9" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 41787 } } }
I20260926 13:49:12.053560 2618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8da52cc844d6430b898a1e5035d40dc9 [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:12.053925 2619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8da52cc844d6430b898a1e5035d40dc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8da52cc844d6430b898a1e5035d40dc9. Latest consensus state: current_term: 1 leader_uuid: "8da52cc844d6430b898a1e5035d40dc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8da52cc844d6430b898a1e5035d40dc9" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 41787 } } }
I20260926 13:49:12.054044 2619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8da52cc844d6430b898a1e5035d40dc9 [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:12.053970 2626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:49:12.057066 2626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:49:12.061996 2626 catalog_manager.cc:1347] Generated new cluster ID: a2ae81fbf31b43a7bb8091c5860cac63
I20260926 13:49:12.062059 2626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:49:12.078084 2626 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:49:12.078891 2626 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:49:12.085422 2626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8da52cc844d6430b898a1e5035d40dc9: Generated new TSK 0
I20260926 13:49:12.085954 2626 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:49:12.093613 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 2539
2026-09-26T13:49:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:49:12 dist-test-slave-hc1k krb5kdc[2638](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:49:12 dist-test-slave-hc1k krb5kdc[2638](info): set up 2 sockets
Sep 26 13:49:12 dist-test-slave-hc1k krb5kdc[2638](info): commencing operation
krb5kdc: starting...
W20260926 13:49:13.039577 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:49:13 dist-test-slave-hc1k krb5kdc[2638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:49:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:49:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:49:13.147840 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:37919
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:38589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:37919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:49:13.249275 2651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:49:13.249497 2651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:49:13.249536 2651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:49:13.253697 2651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:49:13.253752 2651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:49:13.253772 2651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:49:13.253786 2651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:49:13.253810 2651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:49:13.258929 2651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:38589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:37919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:37919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:49:13.259843 2651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:49:13.260761 2651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:49:13.268522 2656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:49:13.268579 2651 server_base.cc:1034] running on GCE node
W20260926 13:49:13.268532 2657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:13.268532 2659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:49:13.269176 2651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:49:13.270174 2651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:49:13.271334 2651 hybrid_clock.cc:648] HybridClock initialized: now 1790430553271321 us; error 28 us; skew 500 ppm
Sep 26 13:49:13 dist-test-slave-hc1k krb5kdc[2638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430553, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:49:13.273808 2651 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:49:13.274561 2651 webserver.cc:460] Webserver started at http://127.12.170.126:38201/ using document root <none> and password file <none>
I20260926 13:49:13.275040 2651 fs_manager.cc:359] Metadata directory not provided
I20260926 13:49:13.275100 2651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:49:13.275267 2651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:49:13.277182 2651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "0d4aa9250768425a9318d22775b20001"
format_stamp: "Formatted at 2026-09-26 13:49:13 on dist-test-slave-hc1k"
server_key: "9746cb945abe4fddc5a9180f1584974d"
server_key_iv: "fa3fddb5f4dd24e5f6b7ba8a49d3375a"
server_key_version: "encryptionkey@0"
I20260926 13:49:13.277674 2651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "0d4aa9250768425a9318d22775b20001"
format_stamp: "Formatted at 2026-09-26 13:49:13 on dist-test-slave-hc1k"
server_key: "9746cb945abe4fddc5a9180f1584974d"
server_key_iv: "fa3fddb5f4dd24e5f6b7ba8a49d3375a"
server_key_version: "encryptionkey@0"
I20260926 13:49:13.281046 2651 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 13:49:13.283178 2666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:49:13.284263 2651 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:49:13.284366 2651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "0d4aa9250768425a9318d22775b20001"
format_stamp: "Formatted at 2026-09-26 13:49:13 on dist-test-slave-hc1k"
server_key: "9746cb945abe4fddc5a9180f1584974d"
server_key_iv: "fa3fddb5f4dd24e5f6b7ba8a49d3375a"
server_key_version: "encryptionkey@0"
I20260926 13:49:13.284452 2651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:49:13.288489 2651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:49:13.290947 2651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:49:13.291093 2651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:49:13.297503 2651 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:37919
I20260926 13:49:13.297520 2718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:37919 every 8 connection(s)
I20260926 13:49:13.298295 2651 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:49:13.300901 2719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:49:13.302181 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 2651
I20260926 13:49:13.302309 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:49:13.302558 12969 external_mini_cluster.cc:1428] Setting key bd6ce1be709465f7ef8332253faebd67
I20260926 13:49:13.306664 2719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d4aa9250768425a9318d22775b20001: Bootstrap starting.
I20260926 13:49:13.307124 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:13.303729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34581 (local address 127.12.170.126:37919)
0926 13:49:13.304100 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:13.304113 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:13.304146 (+ 33us) server_negotiation.cc:407] Connection header received
0926 13:49:13.304734 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:13.304749 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:13.305027 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:13.305278 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:13.306040 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:13.306044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:13.306055 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:49:13.306064 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:13.306114 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:49:13.306131 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:13.306227 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:13.306319 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 13:49:13.306476 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":107,"threads_started":1}
I20260926 13:49:13.308935 2719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d4aa9250768425a9318d22775b20001: Neither blocks nor log segments found. Creating new log.
I20260926 13:49:13.309671 2719 log.cc:824] T 00000000000000000000000000000000 P 0d4aa9250768425a9318d22775b20001: Log is configured to *not* fsync() on all Append() calls
I20260926 13:49:13.309731 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:49:13.311359 2719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d4aa9250768425a9318d22775b20001: No bootstrap required, opened a new log
I20260926 13:49:13.312964 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:13.311510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45213 (local address 127.12.170.126:37919)
0926 13:49:13.311617 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:13.311621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:13.311808 (+ 187us) server_negotiation.cc:407] Connection header received
0926 13:49:13.311909 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:13.311911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:13.311940 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:13.311998 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:13.312604 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:13.312607 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:13.312610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:49:13.312651 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:13.312674 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:49:13.312684 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:13.312755 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:13.312830 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 13:49:13.312864 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 13:49:13.314913 2719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d4aa9250768425a9318d22775b20001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d4aa9250768425a9318d22775b20001" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 37919 } }
I20260926 13:49:13.315111 2719 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d4aa9250768425a9318d22775b20001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:49:13.315157 2719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d4aa9250768425a9318d22775b20001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d4aa9250768425a9318d22775b20001, State: Initialized, Role: FOLLOWER
I20260926 13:49:13.315522 2719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d4aa9250768425a9318d22775b20001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d4aa9250768425a9318d22775b20001" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 37919 } }
I20260926 13:49:13.315634 2719 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d4aa9250768425a9318d22775b20001 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:49:13.315711 2719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d4aa9250768425a9318d22775b20001 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:49:13.315800 2719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d4aa9250768425a9318d22775b20001 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:49:13.316540 2719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d4aa9250768425a9318d22775b20001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d4aa9250768425a9318d22775b20001" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 37919 } }
I20260926 13:49:13.316833 2719 leader_election.cc:302] T 00000000000000000000000000000000 P 0d4aa9250768425a9318d22775b20001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d4aa9250768425a9318d22775b20001; no voters:
I20260926 13:49:13.317082 2719 leader_election.cc:288] T 00000000000000000000000000000000 P 0d4aa9250768425a9318d22775b20001 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:49:13.317214 2729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d4aa9250768425a9318d22775b20001 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:49:13.317391 2729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d4aa9250768425a9318d22775b20001 [term 1 LEADER]: Becoming Leader. State: Replica: 0d4aa9250768425a9318d22775b20001, State: Running, Role: LEADER
I20260926 13:49:13.317668 2729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d4aa9250768425a9318d22775b20001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d4aa9250768425a9318d22775b20001" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 37919 } }
I20260926 13:49:13.318123 2719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d4aa9250768425a9318d22775b20001 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:49:13.319133 2731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d4aa9250768425a9318d22775b20001 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d4aa9250768425a9318d22775b20001. Latest consensus state: current_term: 1 leader_uuid: "0d4aa9250768425a9318d22775b20001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d4aa9250768425a9318d22775b20001" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 37919 } } }
I20260926 13:49:13.319132 2730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d4aa9250768425a9318d22775b20001 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d4aa9250768425a9318d22775b20001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d4aa9250768425a9318d22775b20001" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 37919 } } }
I20260926 13:49:13.319262 2731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d4aa9250768425a9318d22775b20001 [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:13.319275 2730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d4aa9250768425a9318d22775b20001 [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:13.319617 2738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:49:13.322525 2738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:49:13.327446 2738 catalog_manager.cc:1347] Generated new cluster ID: baa5183054cc4bc29ae190c04f3d8587
I20260926 13:49:13.327502 2738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:49:13.347222 2738 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:49:13.347486 2738 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:49:13.350835 2738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d4aa9250768425a9318d22775b20001: Generated new TSK 0
I20260926 13:49:13.351405 2738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:49:13.355976 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 2651
2026-09-26T13:49:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1262 ms)
[----------] 18 tests from AuthTokenIssuingTest (23098 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:49:13 dist-test-slave-hc1k krb5kdc[2750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:49:13 dist-test-slave-hc1k krb5kdc[2750](info): set up 2 sockets
Sep 26 13:49:13 dist-test-slave-hc1k krb5kdc[2750](info): commencing operation
krb5kdc: starting...
W20260926 13:49:14.295586 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:49:14 dist-test-slave-hc1k krb5kdc[2750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:49:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:49:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:49:14.404385 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:40155
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:38943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:40155,127.12.170.125:44205,127.12.170.124:38213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:49:14.506361 2763 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 13:49:14.506608 2763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:49:14.506639 2763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:49:14.506693 2763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:49:14.510556 2763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:49:14.510605 2763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:49:14.510632 2763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:49:14.510653 2763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:49:14.510677 2763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:49:14.515615 2763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:38943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:40155,127.12.170.125:44205,127.12.170.124:38213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:40155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:49:14.516497 2763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:49:14.517365 2763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:49:14.524910 2769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:14.524927 2771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:14.524941 2768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:49:14.525067 2763 server_base.cc:1034] running on GCE node
I20260926 13:49:14.525727 2763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:49:14.526708 2763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:49:14.527899 2763 hybrid_clock.cc:648] HybridClock initialized: now 1790430554527879 us; error 31 us; skew 500 ppm
Sep 26 13:49:14 dist-test-slave-hc1k krb5kdc[2750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430554, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:49:14.530350 2763 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:49:14.531102 2763 webserver.cc:460] Webserver started at http://127.12.170.126:42689/ using document root <none> and password file <none>
I20260926 13:49:14.531602 2763 fs_manager.cc:359] Metadata directory not provided
I20260926 13:49:14.531689 2763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:49:14.531883 2763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:49:14.533649 2763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "91356a680f9c4d5a9ca5eee9786f09df"
format_stamp: "Formatted at 2026-09-26 13:49:14 on dist-test-slave-hc1k"
server_key: "3bdb8ec3ed31c2a3a4c0b46c6e6014d3"
server_key_iv: "e182718f81ae6c736f4d0a932631e9ab"
server_key_version: "encryptionkey@0"
I20260926 13:49:14.534140 2763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "91356a680f9c4d5a9ca5eee9786f09df"
format_stamp: "Formatted at 2026-09-26 13:49:14 on dist-test-slave-hc1k"
server_key: "3bdb8ec3ed31c2a3a4c0b46c6e6014d3"
server_key_iv: "e182718f81ae6c736f4d0a932631e9ab"
server_key_version: "encryptionkey@0"
I20260926 13:49:14.537371 2763 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:49:14.539471 2778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:49:14.540534 2763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:49:14.540635 2763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "91356a680f9c4d5a9ca5eee9786f09df"
format_stamp: "Formatted at 2026-09-26 13:49:14 on dist-test-slave-hc1k"
server_key: "3bdb8ec3ed31c2a3a4c0b46c6e6014d3"
server_key_iv: "e182718f81ae6c736f4d0a932631e9ab"
server_key_version: "encryptionkey@0"
I20260926 13:49:14.540715 2763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:49:14.549865 2763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:49:14.552479 2763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:49:14.552625 2763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:49:14.558501 2763 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:40155
I20260926 13:49:14.558719 2830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:40155 every 8 connection(s)
I20260926 13:49:14.559307 2763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:49:14.561779 2831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:49:14.565110 2831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40155 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:49:14.569152 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 2763
I20260926 13:49:14.569300 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:49:14.569588 12969 external_mini_cluster.cc:1428] Setting key 11f1a4e9c71be8898eea9e46444a3ef9
Sep 26 13:49:14 dist-test-slave-hc1k krb5kdc[2750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430554, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.170.125@KRBTEST.COM; defaulting to no policy
I20260926 13:49:14.579457 2834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.567011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55072 (local address 127.12.170.126:40155)
0926 13:49:14.567387 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:14.567401 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:14.567573 (+ 172us) server_negotiation.cc:407] Connection header received
0926 13:49:14.568553 (+ 980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:14.568558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:14.568722 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:14.568873 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:14.569856 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.570684 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.572053 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.572529 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.572670 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:14.574790 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:14.574797 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:14.574811 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:14.574846 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:14.576674 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:14.577165 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:14.577171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:14.577173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:14.577294 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:14.577584 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:14.577587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:14.577589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:14.577976 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:14.578074 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:14.578492 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 13:49:14.578660 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"spinlock_wait_cycles":36608,"thread_start_us":124,"threads_started":1}
I20260926 13:49:14.579460 2833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.567012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40155 (local address 127.0.0.1:55072)
0926 13:49:14.567436 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:14.567480 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 13:49:14.568266 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:49:14.568919 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:49:14.568926 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:49:14.569378 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:49:14.569703 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:14.569721 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.570807 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.570810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:14.571948 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:14.571960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.572662 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.572665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:14.572742 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:14.573253 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:49:14.573268 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:49:14.574650 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:49:14.576808 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:14.576816 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:14.576824 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:14.577054 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:14.577377 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:14.577379 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:14.577380 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:14.577487 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:14.578072 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:49:14.578077 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:49:14.578201 (+ 124us) client_negotiation.cc:769] Sending connection context
0926 13:49:14.578399 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 13:49:14.578745 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":77,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.125@KRBTEST.COM" created.
I20260926 13:49:14.581831 2831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.125" port: 44205 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:49:14.582513 2833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.582252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:44205 (local address 127.0.0.1:52504)
0926 13:49:14.582392 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:14.582461 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.125:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 13:49:14.582698 2779 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.170.125:44205: connect: Connection refused (error 111)
I20260926 13:49:14.583480 2833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.583287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:44205 (local address 127.0.0.1:52518)
0926 13:49:14.583386 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:14.583431 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.125:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 13:49:14.583628 2831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.125:44205: Network error: Client connection negotiation failed: client connection to 127.12.170.125:44205: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.125.keytab.
Entry for principal kudu/127.12.170.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.125.keytab.
I20260926 13:49:14.594295 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.125:44205
--webserver_interface=127.12.170.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.125.keytab
--principal=kudu/127.12.170.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:38943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:40155,127.12.170.125:44205,127.12.170.124:38213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:49:14.632930 2831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.125" port: 44205 } attempt: 1
I20260926 13:49:14.633778 2833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.633496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:44205 (local address 127.0.0.1:52526)
0926 13:49:14.633640 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:14.633700 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.125:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 13:49:14.634480 2833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.634269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:44205 (local address 127.0.0.1:52536)
0926 13:49:14.634375 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:14.634422 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.125:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 13:49:14.634586 2831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.125:44205: Network error: Client connection negotiation failed: client connection to 127.12.170.125:44205: connect: Connection refused (error 111)
W20260926 13:49:14.695962 2838 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 13:49:14.696203 2838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:49:14.696234 2838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:49:14.696293 2838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:49:14.700124 2838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:49:14.700169 2838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:49:14.700193 2838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:49:14.700222 2838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:49:14.700250 2838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:49:14.702746 2831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.125" port: 44205 } attempt: 2
I20260926 13:49:14.703539 2833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.703229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:44205 (local address 127.0.0.1:52542)
0926 13:49:14.703399 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:14.703460 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.125:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 13:49:14.704312 2833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.704062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:44205 (local address 127.0.0.1:52552)
0926 13:49:14.704173 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:14.704226 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.125:44205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 13:49:14.704433 2831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.125:44205: Network error: Client connection negotiation failed: client connection to 127.12.170.125:44205: connect: Connection refused (error 111)
I20260926 13:49:14.705561 2838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:38943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:40155,127.12.170.125:44205,127.12.170.124:38213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.125.keytab
--principal=kudu/127.12.170.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.125:44205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.170.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:49:14.706432 2838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:49:14.707285 2838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:49:14.714819 2843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:14.714807 2844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:49:14.714975 2838 server_base.cc:1034] running on GCE node
W20260926 13:49:14.714826 2846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:49:14.715538 2838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:49:14.716558 2838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:49:14.717702 2838 hybrid_clock.cc:648] HybridClock initialized: now 1790430554717686 us; error 25 us; skew 500 ppm
Sep 26 13:49:14 dist-test-slave-hc1k krb5kdc[2750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430554, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:49:14.720230 2838 init.cc:377] Logged in from keytab as kudu/127.12.170.125@KRBTEST.COM (short username kudu)
I20260926 13:49:14.721024 2838 webserver.cc:460] Webserver started at http://127.12.170.125:43493/ using document root <none> and password file <none>
I20260926 13:49:14.721621 2838 fs_manager.cc:359] Metadata directory not provided
I20260926 13:49:14.721688 2838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:49:14.721861 2838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:49:14.723699 2838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-1/data/instance:
uuid: "27dd313d415f45b3b75e380bb1e8bcca"
format_stamp: "Formatted at 2026-09-26 13:49:14 on dist-test-slave-hc1k"
server_key: "eedff80f1fdbf4b0afb6587071fb175b"
server_key_iv: "254b154ab88a6cea27076d2d48e14618"
server_key_version: "encryptionkey@0"
I20260926 13:49:14.724191 2838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-1/wal/instance:
uuid: "27dd313d415f45b3b75e380bb1e8bcca"
format_stamp: "Formatted at 2026-09-26 13:49:14 on dist-test-slave-hc1k"
server_key: "eedff80f1fdbf4b0afb6587071fb175b"
server_key_iv: "254b154ab88a6cea27076d2d48e14618"
server_key_version: "encryptionkey@0"
I20260926 13:49:14.727420 2838 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 13:49:14.729607 2853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:49:14.730696 2838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:49:14.730798 2838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-1/wal
uuid: "27dd313d415f45b3b75e380bb1e8bcca"
format_stamp: "Formatted at 2026-09-26 13:49:14 on dist-test-slave-hc1k"
server_key: "eedff80f1fdbf4b0afb6587071fb175b"
server_key_iv: "254b154ab88a6cea27076d2d48e14618"
server_key_version: "encryptionkey@0"
I20260926 13:49:14.730880 2838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:49:14.747635 2838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:49:14.750068 2838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:49:14.750221 2838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:49:14.756399 2838 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.125:44205
I20260926 13:49:14.756631 2905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.125:44205 every 8 connection(s)
I20260926 13:49:14.757211 2838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-1/data/info.pb
I20260926 13:49:14.759294 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 2838
I20260926 13:49:14.759428 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-1/wal/instance
I20260926 13:49:14.759794 12969 external_mini_cluster.cc:1428] Setting key c4f5d22535f1de9a859c725a5bd13d71
I20260926 13:49:14.759892 2906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:49:14.763782 2906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40155 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.170.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.124@KRBTEST.COM" created.
Sep 26 13:49:14 dist-test-slave-hc1k krb5kdc[2750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430554, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.125@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:49:14.776696 2834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.765506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55078 (local address 127.12.170.126:40155)
0926 13:49:14.765655 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:14.765661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:14.766412 (+ 751us) server_negotiation.cc:407] Connection header received
0926 13:49:14.767308 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:14.767311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:14.767362 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:14.767433 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:14.768699 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.769402 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.770901 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.771239 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.771351 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:14.773545 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:14.773548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:14.773550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:14.773574 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:14.774844 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:14.775296 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:14.775299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:14.775300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:14.775373 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:14.775687 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:14.775690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:14.775692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:14.775872 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:14.775957 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:14.776419 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 13:49:14.776536 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 13:49:14.777501 2909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.765793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40155 (local address 127.0.0.1:55078)
0926 13:49:14.766234 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:14.766295 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 13:49:14.767120 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:49:14.767637 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:49:14.767643 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:49:14.767992 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:49:14.768565 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:14.768580 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.769529 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.769533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:14.770779 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:14.770789 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.771398 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.771401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:14.771531 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:14.772066 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:49:14.772082 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:49:14.773392 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:49:14.774955 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:14.774960 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:14.774974 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:14.775206 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:14.775452 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:14.775455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:14.775457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:14.775582 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:14.775968 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:49:14.775974 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:49:14.776193 (+ 219us) client_negotiation.cc:769] Sending connection context
0926 13:49:14.776383 (+ 190us) client_negotiation.cc:240] Negotiation successful
0926 13:49:14.776665 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":108,"threads_started":1}
I20260926 13:49:14.778676 2906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.125" port: 44205 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.124.keytab.
Entry for principal kudu/127.12.170.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.124.keytab.
I20260926 13:49:14.783212 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.124:38213
--webserver_interface=127.12.170.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.124.keytab
--principal=kudu/127.12.170.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:38943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:40155,127.12.170.125:44205,127.12.170.124:38213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:49:14 dist-test-slave-hc1k krb5kdc[2750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430554, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.125@KRBTEST.COM for kudu/127.12.170.125@KRBTEST.COM
I20260926 13:49:14.786626 2909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.779125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:44205 (local address 127.0.0.1:52558)
0926 13:49:14.779224 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:14.779235 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:49:14.779294 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:49:14.779873 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:49:14.779875 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:49:14.780012 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:49:14.780089 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:14.780093 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.780961 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.780963 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:14.781552 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:14.781560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.782059 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.782061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:14.782160 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:14.782605 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:49:14.782619 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:49:14.783846 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:49:14.785372 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:14.785374 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:14.785377 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:14.785541 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:14.785771 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:14.785774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:14.785775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:14.785850 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:14.786206 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:49:14.786208 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:49:14.786285 (+ 77us) client_negotiation.cc:769] Sending connection context
0926 13:49:14.786370 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 13:49:14.786477 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":24}
I20260926 13:49:14.786724 2911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.779232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52558 (local address 127.12.170.125:44205)
0926 13:49:14.779488 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:14.779498 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:14.779520 (+ 22us) server_negotiation.cc:407] Connection header received
0926 13:49:14.779561 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:14.779567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:14.779770 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:14.779877 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:14.780173 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.780864 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.781640 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.781978 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.782054 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:14.783954 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:14.783961 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:14.783970 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:14.783988 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:14.785280 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:14.785627 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:14.785631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:14.785632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:14.785697 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:14.785919 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:14.785922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:14.785924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:14.786116 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:14.786222 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:14.786449 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 13:49:14.786558 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":173,"thread_start_us":66,"threads_started":1}
I20260926 13:49:14.788398 2906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:49:14.788940 2909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.788735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:38213 (local address 127.0.0.1:38736)
0926 13:49:14.788826 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:14.788891 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 13:49:14.789108 2855 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.170.124:38213: connect: Connection refused (error 111)
I20260926 13:49:14.789765 2909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.789594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:38213 (local address 127.0.0.1:38748)
0926 13:49:14.789679 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:14.789721 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 13:49:14.789883 2906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.124:38213: Network error: Client connection negotiation failed: client connection to 127.12.170.124:38213: connect: Connection refused (error 111)
I20260926 13:49:14.795579 2831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.125" port: 44205 } attempt: 3
Sep 26 13:49:14 dist-test-slave-hc1k krb5kdc[2750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430554, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for kudu/127.12.170.125@KRBTEST.COM
I20260926 13:49:14.803371 2833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.796062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:44205 (local address 127.0.0.1:52566)
0926 13:49:14.796206 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:14.796218 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:49:14.796300 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:49:14.796488 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:49:14.796491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:49:14.796634 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:49:14.796719 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:14.796724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.797388 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.797391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:14.798279 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:14.798288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.799100 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.799104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:14.799197 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:14.799570 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:49:14.799583 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:49:14.800686 (+ 1103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:49:14.802128 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:14.802131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:14.802133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:14.802304 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:14.802532 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:14.802535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:14.802537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:14.802608 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:14.802914 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:49:14.802918 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:49:14.803004 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 13:49:14.803086 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 13:49:14.803211 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 13:49:14.803369 2911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.796067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52566 (local address 127.12.170.125:44205)
0926 13:49:14.796154 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:14.796158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:14.796268 (+ 110us) server_negotiation.cc:407] Connection header received
0926 13:49:14.796365 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:14.796368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:14.796405 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:14.796465 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:14.796814 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.797275 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.798386 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.798998 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.799086 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:14.800798 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:14.800802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:14.800804 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:14.800827 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:14.802034 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:14.802385 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:14.802388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:14.802390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:14.802458 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:14.802681 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:14.802684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:14.802686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:14.802834 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:14.802911 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:14.803141 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 13:49:14.803228 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 13:49:14.804049 2831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:49:14.804628 2833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.804434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:38213 (local address 127.0.0.1:38752)
0926 13:49:14.804532 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:14.804575 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 13:49:14.805240 2833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.805027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:38213 (local address 127.0.0.1:38768)
0926 13:49:14.805145 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:14.805189 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260926 13:49:14.805377 2831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.124:38213: Network error: Client connection negotiation failed: client connection to 127.12.170.124:38213: connect: Connection refused (error 111)
I20260926 13:49:14.836550 2831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } attempt: 1
I20260926 13:49:14.837141 2833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.836923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:38213 (local address 127.0.0.1:38770)
0926 13:49:14.837036 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:14.837082 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 13:49:14.837743 2833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.837576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:38213 (local address 127.0.0.1:38772)
0926 13:49:14.837655 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:14.837693 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 13:49:14.837858 2831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.124:38213: Network error: Client connection negotiation failed: client connection to 127.12.170.124:38213: connect: Connection refused (error 111)
I20260926 13:49:14.839075 2906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } attempt: 1
I20260926 13:49:14.839591 2909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.839396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:38213 (local address 127.0.0.1:38784)
0926 13:49:14.839495 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:14.839538 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 13:49:14.840147 2909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.839977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:38213 (local address 127.0.0.1:38788)
0926 13:49:14.840061 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:14.840099 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 13:49:14.840241 2906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.124:38213: Network error: Client connection negotiation failed: client connection to 127.12.170.124:38213: connect: Connection refused (error 111)
W20260926 13:49:14.886154 2912 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 13:49:14.886405 2912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:49:14.886439 2912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:49:14.886498 2912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:49:14.890393 2912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:49:14.890444 2912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:49:14.890470 2912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:49:14.890491 2912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:49:14.890515 2912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:49:14.895473 2912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:38943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:40155,127.12.170.125:44205,127.12.170.124:38213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.124.keytab
--principal=kudu/127.12.170.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.124:38213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.170.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:49:14.896358 2912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:49:14.897235 2912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:49:14.904772 2921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:14.904745 2919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:14.904762 2918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:49:14.904963 2912 server_base.cc:1034] running on GCE node
I20260926 13:49:14.905494 2912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:49:14.906446 2912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:49:14.907591 2912 hybrid_clock.cc:648] HybridClock initialized: now 1790430554907571 us; error 30 us; skew 500 ppm
I20260926 13:49:14.908377 2906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } attempt: 2
I20260926 13:49:14.909016 2909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.908772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:38213 (local address 127.0.0.1:38800)
0926 13:49:14.908896 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:14.908947 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 13:49:14.909603 2909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.909404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:38213 (local address 127.0.0.1:38804)
0926 13:49:14.909500 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:14.909544 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 13:49:14.909736 2906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.124:38213: Network error: Client connection negotiation failed: client connection to 127.12.170.124:38213: connect: Connection refused (error 111)
Sep 26 13:49:14 dist-test-slave-hc1k krb5kdc[2750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430554, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:49:14.910195 2912 init.cc:377] Logged in from keytab as kudu/127.12.170.124@KRBTEST.COM (short username kudu)
I20260926 13:49:14.910945 2912 webserver.cc:460] Webserver started at http://127.12.170.124:33383/ using document root <none> and password file <none>
I20260926 13:49:14.911419 2912 fs_manager.cc:359] Metadata directory not provided
I20260926 13:49:14.911473 2912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:49:14.911674 2912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:49:14.913002 2831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } attempt: 2
I20260926 13:49:14.913618 2833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.913388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:38213 (local address 127.0.0.1:38818)
0926 13:49:14.913507 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:14.913557 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 13:49:14.913655 2912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-2/data/instance:
uuid: "958ad1e870f14b83bed17afe167e71e9"
format_stamp: "Formatted at 2026-09-26 13:49:14 on dist-test-slave-hc1k"
server_key: "a8c55860dfcb4b545ec5aa048fa5a667"
server_key_iv: "060a372cc19f14a2c9f62c7e506ef254"
server_key_version: "encryptionkey@0"
I20260926 13:49:14.914232 2833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.914034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:38213 (local address 127.0.0.1:38834)
0926 13:49:14.914129 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:14.914175 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:38213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 13:49:14.914232 2912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-2/wal/instance:
uuid: "958ad1e870f14b83bed17afe167e71e9"
format_stamp: "Formatted at 2026-09-26 13:49:14 on dist-test-slave-hc1k"
server_key: "a8c55860dfcb4b545ec5aa048fa5a667"
server_key_iv: "060a372cc19f14a2c9f62c7e506ef254"
server_key_version: "encryptionkey@0"
W20260926 13:49:14.914371 2831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.124:38213: Network error: Client connection negotiation failed: client connection to 127.12.170.124:38213: connect: Connection refused (error 111)
I20260926 13:49:14.917546 2912 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:49:14.919615 2928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:49:14.920710 2912 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:49:14.920804 2912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-2/wal
uuid: "958ad1e870f14b83bed17afe167e71e9"
format_stamp: "Formatted at 2026-09-26 13:49:14 on dist-test-slave-hc1k"
server_key: "a8c55860dfcb4b545ec5aa048fa5a667"
server_key_iv: "060a372cc19f14a2c9f62c7e506ef254"
server_key_version: "encryptionkey@0"
I20260926 13:49:14.920900 2912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:49:14.940197 2912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:49:14.942716 2912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:49:14.942874 2912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:49:14.949047 2912 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.124:38213
I20260926 13:49:14.949325 2980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.124:38213 every 8 connection(s)
I20260926 13:49:14.949887 2912 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-2/data/info.pb
I20260926 13:49:14.952433 2981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:49:14.955984 2981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40155 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:49:14.958006 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 2912
I20260926 13:49:14.958137 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790430291939677-12969-0/minicluster-data/master-2/wal/instance
I20260926 13:49:14.958415 12969 external_mini_cluster.cc:1428] Setting key 82ef724af5e1617e74ef802ea58f8c4d
Sep 26 13:49:14 dist-test-slave-hc1k krb5kdc[2750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430554, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.124@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
Sep 26 13:49:14 dist-test-slave-hc1k krb5kdc[2750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.124@KRBTEST.COM
I20260926 13:49:14.968209 2834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.957646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55084 (local address 127.12.170.126:40155)
0926 13:49:14.957753 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:14.957756 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:14.958592 (+ 836us) server_negotiation.cc:407] Connection header received
0926 13:49:14.959612 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:14.959616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:14.959693 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:14.959763 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:14.960647 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.961133 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.962042 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.962251 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.962345 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:14.964738 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:14.964741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:14.964743 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:14.964766 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:14.966243 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:14.966711 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:14.966714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:14.966716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:14.966802 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:14.967138 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:14.967140 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:14.967142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:14.967387 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:14.967456 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:14.967953 (+ 497us) server_negotiation.cc:299] Negotiation successful
0926 13:49:14.968056 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 13:49:14.969009 2983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.957996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:40155 (local address 127.0.0.1:55084)
0926 13:49:14.958440 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:14.958488 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 13:49:14.959420 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:49:14.959860 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:49:14.959869 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:49:14.960180 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:49:14.960517 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:14.960528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.961225 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.961229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:14.961934 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:14.961943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.962374 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.962378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:14.962529 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:14.963020 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:49:14.963036 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:49:14.964608 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:49:14.966364 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:14.966371 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:14.966378 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:14.966624 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:14.966886 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:14.966888 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:14.966890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:14.967049 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:14.967478 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:49:14.967483 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:49:14.967741 (+ 258us) client_negotiation.cc:769] Sending connection context
0926 13:49:14.967895 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 13:49:14.968172 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"spinlock_wait_cycles":1920,"thread_start_us":128,"threads_started":1}
I20260926 13:49:14.969007 2984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.958991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38850 (local address 127.12.170.124:38213)
0926 13:49:14.959260 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:14.959273 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:14.959478 (+ 205us) server_negotiation.cc:407] Connection header received
0926 13:49:14.959693 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:14.959698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:14.959863 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:14.960009 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:14.960770 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.961581 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.962358 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.962677 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.962762 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:14.965345 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:14.965352 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:14.965363 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:14.965387 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:14.966812 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:14.967217 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:14.967222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:14.967223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:14.967298 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:14.967536 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:14.967538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:14.967539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:14.967798 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:14.967976 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:14.968169 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 13:49:14.968488 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"spinlock_wait_cycles":4480,"thread_start_us":91,"threads_started":1}
I20260926 13:49:14.970090 2981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.125" port: 44205 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:49:14 dist-test-slave-hc1k krb5kdc[2750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430554, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.124@KRBTEST.COM for kudu/127.12.170.125@KRBTEST.COM
Sep 26 13:49:14 dist-test-slave-hc1k krb5kdc[2750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:49:14.977607 2983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.970470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:44205 (local address 127.0.0.1:52574)
0926 13:49:14.970613 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:14.970626 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:49:14.970682 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:49:14.970856 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:49:14.970858 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:49:14.970967 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:49:14.971043 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:14.971047 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.971821 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.971824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:14.972576 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:14.972586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.973103 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.973105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:14.973176 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:14.973598 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:49:14.973611 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:49:14.974757 (+ 1146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:49:14.976229 (+ 1472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:14.976232 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:14.976234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:14.976460 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:14.976710 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:14.976713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:14.976715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:14.976801 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:14.977174 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:49:14.977177 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:49:14.977257 (+ 80us) client_negotiation.cc:769] Sending connection context
0926 13:49:14.977343 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 13:49:14.977444 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260926 13:49:14.977711 2911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.970585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52574 (local address 127.12.170.125:44205)
0926 13:49:14.970697 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:14.970700 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:14.970712 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:49:14.970747 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:14.970749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:14.970784 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:14.970845 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:14.971139 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.971727 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.972689 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.973004 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.973114 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:14.974884 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:14.974888 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:14.974890 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:14.974917 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:14.976135 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:14.976550 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:14.976552 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:14.976554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:14.976627 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:14.976880 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:14.976884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:14.976886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:14.977077 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:14.977169 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:14.977448 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 13:49:14.977551 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 13:49:14.978384 2981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:49:14.979645 2834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.971289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55094 (local address 127.12.170.126:40155)
0926 13:49:14.971890 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:14.971894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:14.971905 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:49:14.971934 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:14.971937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:14.971972 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:14.972037 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:14.972751 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.973398 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.974358 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.974617 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.974701 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:14.976782 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:14.976784 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:14.976786 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:14.976804 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:14.978161 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:14.978598 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:14.978601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:14.978603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:14.978678 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:14.978931 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:14.978934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:14.978935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:14.979101 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:14.979300 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:14.979412 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 13:49:14.979511 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530}
Sep 26 13:49:14 dist-test-slave-hc1k krb5kdc[2750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430554, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.124@KRBTEST.COM for kudu/127.12.170.124@KRBTEST.COM
Sep 26 13:49:14 dist-test-slave-hc1k krb5kdc[2750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.125@KRBTEST.COM
I20260926 13:49:14.985759 2983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.978756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:38213 (local address 127.0.0.1:38858)
0926 13:49:14.978863 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:14.978876 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:49:14.978939 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:49:14.979191 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:49:14.979194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:49:14.979357 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:49:14.979434 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:14.979440 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.980270 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.980273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:14.980878 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:14.980886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.981280 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.981283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:14.981399 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:14.981747 (+ 348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:49:14.981759 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:49:14.982986 (+ 1227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:49:14.984561 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:14.984563 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:14.984565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:14.984730 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:14.984960 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:14.984963 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:14.984964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:14.985052 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:14.985343 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:49:14.985346 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:49:14.985421 (+ 75us) client_negotiation.cc:769] Sending connection context
0926 13:49:14.985520 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 13:49:14.985624 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":14}
I20260926 13:49:14.985947 2984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.978789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38858 (local address 127.12.170.124:38213)
0926 13:49:14.979045 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:14.979047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:14.979054 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:49:14.979073 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:14.979074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:14.979099 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:14.979146 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:14.979544 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.980173 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.980973 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.981198 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.981275 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:14.983101 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:14.983105 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:14.983107 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:14.983131 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:14.984467 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:14.984817 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:14.984820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:14.984822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:14.984889 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:14.985118 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:14.985121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:14.985122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:14.985266 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:14.985349 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:14.985606 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 13:49:14.985713 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":210}
I20260926 13:49:14.988726 2911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:14.980656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52580 (local address 127.12.170.125:44205)
0926 13:49:14.980751 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:14.980754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:14.980767 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:49:14.980802 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:14.980805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:14.980840 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:14.980898 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:14.981409 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.981840 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.982746 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:14.983052 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:14.983141 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:14.985433 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:14.985437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:14.985439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:14.985461 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:14.987127 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:14.987499 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:14.987503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:14.987506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:14.987605 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:14.987934 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:14.987937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:14.987939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:14.988148 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:14.988390 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:14.988471 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 13:49:14.988576 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 13:49:14.989511 2981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 958ad1e870f14b83bed17afe167e71e9: Bootstrap starting.
I20260926 13:49:14.991693 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:49:14.991837 2981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 958ad1e870f14b83bed17afe167e71e9: Neither blocks nor log segments found. Creating new log.
I20260926 13:49:14.992533 2981 log.cc:824] T 00000000000000000000000000000000 P 958ad1e870f14b83bed17afe167e71e9: Log is configured to *not* fsync() on all Append() calls
I20260926 13:49:14.992818 2805 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "91356a680f9c4d5a9ca5eee9786f09df"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55094
I20260926 13:49:14.994434 2981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 958ad1e870f14b83bed17afe167e71e9: No bootstrap required, opened a new log
I20260926 13:49:14.997408 2981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 958ad1e870f14b83bed17afe167e71e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91356a680f9c4d5a9ca5eee9786f09df" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40155 } } peers { permanent_uuid: "27dd313d415f45b3b75e380bb1e8bcca" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 44205 } } peers { permanent_uuid: "958ad1e870f14b83bed17afe167e71e9" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } }
I20260926 13:49:14.997601 2981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 958ad1e870f14b83bed17afe167e71e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 958ad1e870f14b83bed17afe167e71e9, State: Initialized, Role: FOLLOWER
I20260926 13:49:14.997895 2981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 958ad1e870f14b83bed17afe167e71e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91356a680f9c4d5a9ca5eee9786f09df" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40155 } } peers { permanent_uuid: "27dd313d415f45b3b75e380bb1e8bcca" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 44205 } } peers { permanent_uuid: "958ad1e870f14b83bed17afe167e71e9" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } }
I20260926 13:49:14.998349 2987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 958ad1e870f14b83bed17afe167e71e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91356a680f9c4d5a9ca5eee9786f09df" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40155 } } peers { permanent_uuid: "27dd313d415f45b3b75e380bb1e8bcca" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 44205 } } peers { permanent_uuid: "958ad1e870f14b83bed17afe167e71e9" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } } }
I20260926 13:49:14.998479 2987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 958ad1e870f14b83bed17afe167e71e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:49:14.999006 2981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 958ad1e870f14b83bed17afe167e71e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:49:15.000875 2906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } attempt: 3
W20260926 13:49:15.004153 2998 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 958ad1e870f14b83bed17afe167e71e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:49:15.004228 2998 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:49:15.002553 2912 master_runner.cc:186] Error getting master registration for 127.12.170.126:40155: OK, instance_id { permanent_uuid: "91356a680f9c4d5a9ca5eee9786f09df" instance_seqno: 1790430554556900 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 13:49:15.005796 2912 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 26 13:49:15 dist-test-slave-hc1k krb5kdc[2750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430554, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.125@KRBTEST.COM for kudu/127.12.170.124@KRBTEST.COM
I20260926 13:49:15.009826 2909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:15.001293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:38213 (local address 127.0.0.1:38868)
0926 13:49:15.001478 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:15.001493 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:49:15.001576 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:49:15.001810 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:49:15.001813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:49:15.001998 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:49:15.002097 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:15.002105 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:15.003061 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:15.003064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:15.003837 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:15.003848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:15.004543 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:15.004546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:15.004623 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:15.005113 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:49:15.005127 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:49:15.006599 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:49:15.008299 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:15.008304 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:15.008306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:15.008564 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:15.008776 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:15.008779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:15.008799 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:15.008894 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:15.009326 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:49:15.009329 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:49:15.009425 (+ 96us) client_negotiation.cc:769] Sending connection context
0926 13:49:15.009533 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 13:49:15.009665 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260926 13:49:15.009836 2984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:15.001298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38868 (local address 127.12.170.124:38213)
0926 13:49:15.001459 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:15.001464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:15.001539 (+ 75us) server_negotiation.cc:407] Connection header received
0926 13:49:15.001660 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:15.001662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:15.001699 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:15.001787 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:15.002248 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:15.002940 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:15.003977 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:15.004431 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:15.004519 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:15.006733 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:15.006737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:15.006740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:15.006764 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:15.008203 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:15.008653 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:15.008655 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:15.008657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:15.008718 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:15.009000 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:15.009003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:15.009005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:15.009227 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:15.009315 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:15.009579 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 13:49:15.009684 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:49:15.012997 2906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27dd313d415f45b3b75e380bb1e8bcca: Bootstrap starting.
I20260926 13:49:15.013528 2831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } attempt: 3
I20260926 13:49:15.015826 2906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27dd313d415f45b3b75e380bb1e8bcca: Neither blocks nor log segments found. Creating new log.
I20260926 13:49:15.016686 2906 log.cc:824] T 00000000000000000000000000000000 P 27dd313d415f45b3b75e380bb1e8bcca: Log is configured to *not* fsync() on all Append() calls
Sep 26 13:49:15 dist-test-slave-hc1k krb5kdc[2750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430554, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for kudu/127.12.170.124@KRBTEST.COM
I20260926 13:49:15.019697 2906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27dd313d415f45b3b75e380bb1e8bcca: No bootstrap required, opened a new log
I20260926 13:49:15.022204 2833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:15.013892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:38213 (local address 127.0.0.1:38876)
0926 13:49:15.014058 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:15.014071 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:49:15.014140 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:49:15.014461 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:49:15.014463 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:49:15.014613 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:49:15.014692 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:15.014697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:15.015613 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:15.015617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:15.016462 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:15.016472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:15.016997 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:15.017001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:15.017086 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:15.017507 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:49:15.017522 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:49:15.018935 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:49:15.020800 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:15.020803 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:15.020805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:15.021052 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:15.021314 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:15.021317 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:15.021319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:15.021393 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:15.021748 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:49:15.021751 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:49:15.021839 (+ 88us) client_negotiation.cc:769] Sending connection context
0926 13:49:15.021921 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 13:49:15.022050 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 13:49:15.022204 2984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:15.013940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38876 (local address 127.12.170.124:38213)
0926 13:49:15.014263 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:15.014266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:15.014276 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:49:15.014304 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:15.014307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:15.014342 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:15.014431 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:15.014795 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:15.015515 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:15.016587 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:15.016894 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:15.016974 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:15.019065 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:15.019069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:15.019071 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:15.019110 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:15.020668 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:15.021136 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:15.021139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:15.021141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:15.021228 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:15.021473 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:15.021476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:15.021478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:15.021657 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:15.021737 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:15.021981 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 13:49:15.022066 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261}
I20260926 13:49:15.023537 2906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27dd313d415f45b3b75e380bb1e8bcca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91356a680f9c4d5a9ca5eee9786f09df" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40155 } } peers { permanent_uuid: "27dd313d415f45b3b75e380bb1e8bcca" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 44205 } } peers { permanent_uuid: "958ad1e870f14b83bed17afe167e71e9" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } }
I20260926 13:49:15.023782 2906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27dd313d415f45b3b75e380bb1e8bcca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27dd313d415f45b3b75e380bb1e8bcca, State: Initialized, Role: FOLLOWER
I20260926 13:49:15.024091 2906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27dd313d415f45b3b75e380bb1e8bcca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91356a680f9c4d5a9ca5eee9786f09df" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40155 } } peers { permanent_uuid: "27dd313d415f45b3b75e380bb1e8bcca" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 44205 } } peers { permanent_uuid: "958ad1e870f14b83bed17afe167e71e9" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } }
I20260926 13:49:15.024606 3001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27dd313d415f45b3b75e380bb1e8bcca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91356a680f9c4d5a9ca5eee9786f09df" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40155 } } peers { permanent_uuid: "27dd313d415f45b3b75e380bb1e8bcca" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 44205 } } peers { permanent_uuid: "958ad1e870f14b83bed17afe167e71e9" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } } }
I20260926 13:49:15.024892 3001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27dd313d415f45b3b75e380bb1e8bcca [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:49:15.025134 2831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df: Bootstrap starting.
I20260926 13:49:15.025358 2906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27dd313d415f45b3b75e380bb1e8bcca [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:49:15.027885 2831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df: Neither blocks nor log segments found. Creating new log.
I20260926 13:49:15.028688 2831 log.cc:824] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df: Log is configured to *not* fsync() on all Append() calls
W20260926 13:49:15.030575 3012 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 27dd313d415f45b3b75e380bb1e8bcca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:49:15.030644 3012 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:49:15.031422 2831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df: No bootstrap required, opened a new log
I20260926 13:49:15.028844 2838 master_runner.cc:186] Error getting master registration for 127.12.170.126:40155: OK, instance_id { permanent_uuid: "91356a680f9c4d5a9ca5eee9786f09df" instance_seqno: 1790430554556900 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 13:49:15.031730 2838 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 13:49:15.036089 2831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91356a680f9c4d5a9ca5eee9786f09df" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40155 } } peers { permanent_uuid: "27dd313d415f45b3b75e380bb1e8bcca" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 44205 } } peers { permanent_uuid: "958ad1e870f14b83bed17afe167e71e9" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } }
I20260926 13:49:15.036402 2831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91356a680f9c4d5a9ca5eee9786f09df, State: Initialized, Role: FOLLOWER
I20260926 13:49:15.036872 2831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91356a680f9c4d5a9ca5eee9786f09df" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40155 } } peers { permanent_uuid: "27dd313d415f45b3b75e380bb1e8bcca" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 44205 } } peers { permanent_uuid: "958ad1e870f14b83bed17afe167e71e9" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } }
I20260926 13:49:15.037448 3015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91356a680f9c4d5a9ca5eee9786f09df" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40155 } } peers { permanent_uuid: "27dd313d415f45b3b75e380bb1e8bcca" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 44205 } } peers { permanent_uuid: "958ad1e870f14b83bed17afe167e71e9" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } } }
I20260926 13:49:15.037645 3015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:49:15.038753 2805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 13:49:15.038875 2805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:49:15.039712 2831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:49:15.040194 2805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91356a680f9c4d5a9ca5eee9786f09df" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40155 } } peers { permanent_uuid: "27dd313d415f45b3b75e380bb1e8bcca" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 44205 } } peers { permanent_uuid: "958ad1e870f14b83bed17afe167e71e9" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } }
I20260926 13:49:15.041093 2805 leader_election.cc:288] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df [CANDIDATE]: Term 1 election: Requested vote from peers 27dd313d415f45b3b75e380bb1e8bcca (127.12.170.125:44205), 958ad1e870f14b83bed17afe167e71e9 (127.12.170.124:38213)
I20260926 13:49:15.042250 2880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91356a680f9c4d5a9ca5eee9786f09df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27dd313d415f45b3b75e380bb1e8bcca"
I20260926 13:49:15.042469 2880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27dd313d415f45b3b75e380bb1e8bcca [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:49:15.043694 2880 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 27dd313d415f45b3b75e380bb1e8bcca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91356a680f9c4d5a9ca5eee9786f09df in term 1.
I20260926 13:49:15.045580 2955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91356a680f9c4d5a9ca5eee9786f09df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "958ad1e870f14b83bed17afe167e71e9"
I20260926 13:49:15.045830 2955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 958ad1e870f14b83bed17afe167e71e9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:49:15.047024 2955 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 958ad1e870f14b83bed17afe167e71e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91356a680f9c4d5a9ca5eee9786f09df in term 1.
I20260926 13:49:15.047525 2779 leader_election.cc:302] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27dd313d415f45b3b75e380bb1e8bcca, 91356a680f9c4d5a9ca5eee9786f09df; no voters:
I20260926 13:49:15.047889 3015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:49:15.048094 3015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df [term 1 LEADER]: Becoming Leader. State: Replica: 91356a680f9c4d5a9ca5eee9786f09df, State: Running, Role: LEADER
I20260926 13:49:15.048399 3015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91356a680f9c4d5a9ca5eee9786f09df" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40155 } } peers { permanent_uuid: "27dd313d415f45b3b75e380bb1e8bcca" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 44205 } } peers { permanent_uuid: "958ad1e870f14b83bed17afe167e71e9" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } }
I20260926 13:49:15.050258 3015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91356a680f9c4d5a9ca5eee9786f09df. Latest consensus state: current_term: 1 leader_uuid: "91356a680f9c4d5a9ca5eee9786f09df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91356a680f9c4d5a9ca5eee9786f09df" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40155 } } peers { permanent_uuid: "27dd313d415f45b3b75e380bb1e8bcca" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 44205 } } peers { permanent_uuid: "958ad1e870f14b83bed17afe167e71e9" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } } }
I20260926 13:49:15.050423 3015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:15.051688 2763 master_runner.cc:263] No leader master found from master 91356a680f9c4d5a9ca5eee9786f09df
I20260926 13:49:15.051751 2763 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 13:49:15.053879 2784 tablet.cc:2369] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:49:15.055586 3026 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:49:15.055734 3026 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:49:15.062115 2834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:15.045547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55098 (local address 127.12.170.126:40155)
0926 13:49:15.047285 (+ 1738us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:15.047290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:15.052083 (+ 4793us) server_negotiation.cc:407] Connection header received
0926 13:49:15.052247 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:15.052250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:15.052290 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:15.052336 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:15.053293 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:15.054063 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:15.055214 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:15.055547 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:15.055612 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:15.058695 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:15.058699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:15.058702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:15.058726 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:15.060372 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:15.060846 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:15.060850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:15.060852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:15.060934 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:15.061205 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:15.061208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:15.061210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:15.061400 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:15.061506 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:15.061844 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 13:49:15.061964 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1666}
I20260926 13:49:15.063529 3036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:49:15.064374 3036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:49:15.065717 2911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:15.046274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52590 (local address 127.12.170.125:44205)
0926 13:49:15.046388 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:15.046393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:15.056086 (+ 9693us) server_negotiation.cc:407] Connection header received
0926 13:49:15.056269 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:15.056273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:15.056321 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:15.056373 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:15.057620 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:15.058377 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:15.059526 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:15.059891 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:15.059965 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:15.061659 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:15.061663 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:15.061666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:15.061694 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:15.063623 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:15.064211 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:15.064215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:15.064217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:15.064312 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:15.064703 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:15.064707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:15.064709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:15.064936 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:15.065055 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:15.065437 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 13:49:15.065558 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:49:15.069955 2955 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 958ad1e870f14b83bed17afe167e71e9 [term 1 FOLLOWER]: Refusing update from remote peer 91356a680f9c4d5a9ca5eee9786f09df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:49:15.070596 3033 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df [LEADER]: Connected to new peer: Peer: permanent_uuid: "958ad1e870f14b83bed17afe167e71e9" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:49:15.074374 2987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 958ad1e870f14b83bed17afe167e71e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91356a680f9c4d5a9ca5eee9786f09df. Latest consensus state: current_term: 1 leader_uuid: "91356a680f9c4d5a9ca5eee9786f09df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91356a680f9c4d5a9ca5eee9786f09df" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40155 } } peers { permanent_uuid: "27dd313d415f45b3b75e380bb1e8bcca" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 44205 } } peers { permanent_uuid: "958ad1e870f14b83bed17afe167e71e9" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } } }
I20260926 13:49:15.074491 2987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 958ad1e870f14b83bed17afe167e71e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:49:15.075485 2880 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 27dd313d415f45b3b75e380bb1e8bcca [term 1 FOLLOWER]: Refusing update from remote peer 91356a680f9c4d5a9ca5eee9786f09df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:49:15.076061 3033 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df [LEADER]: Connected to new peer: Peer: permanent_uuid: "27dd313d415f45b3b75e380bb1e8bcca" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 44205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:49:15.077028 2984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:15.047937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38878 (local address 127.12.170.124:38213)
0926 13:49:15.048051 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:15.048055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:15.057452 (+ 9397us) server_negotiation.cc:407] Connection header received
0926 13:49:15.057573 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:15.057576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:15.057621 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:15.057698 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:15.059741 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:15.060462 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:15.071740 (+ 11278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:15.072054 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:15.072126 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:15.073738 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:15.073741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:15.073744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:15.073769 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:15.075271 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:15.075751 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:15.075755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:15.075756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:15.075839 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:15.076137 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:15.076140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:15.076142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:15.076354 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:15.076454 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:15.076785 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 13:49:15.076892 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 13:49:15.080130 3001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27dd313d415f45b3b75e380bb1e8bcca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91356a680f9c4d5a9ca5eee9786f09df. Latest consensus state: current_term: 1 leader_uuid: "91356a680f9c4d5a9ca5eee9786f09df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91356a680f9c4d5a9ca5eee9786f09df" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40155 } } peers { permanent_uuid: "27dd313d415f45b3b75e380bb1e8bcca" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 44205 } } peers { permanent_uuid: "958ad1e870f14b83bed17afe167e71e9" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } } }
I20260926 13:49:15.080246 3001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27dd313d415f45b3b75e380bb1e8bcca [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:49:15.084156 3033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "91356a680f9c4d5a9ca5eee9786f09df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91356a680f9c4d5a9ca5eee9786f09df" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40155 } } peers { permanent_uuid: "27dd313d415f45b3b75e380bb1e8bcca" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 44205 } } peers { permanent_uuid: "958ad1e870f14b83bed17afe167e71e9" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } } }
I20260926 13:49:15.084264 3033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:15.085322 3001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27dd313d415f45b3b75e380bb1e8bcca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "91356a680f9c4d5a9ca5eee9786f09df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91356a680f9c4d5a9ca5eee9786f09df" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40155 } } peers { permanent_uuid: "27dd313d415f45b3b75e380bb1e8bcca" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 44205 } } peers { permanent_uuid: "958ad1e870f14b83bed17afe167e71e9" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } } }
I20260926 13:49:15.085421 3001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27dd313d415f45b3b75e380bb1e8bcca [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:49:15.085741 3038 mvcc.cc:205] Tried to move back new op lower bound from 7333603553560408064 to 7333603553482121216. Current Snapshot: MvccSnapshot[applied={T|T < 7333603553560408064}]
I20260926 13:49:15.088603 3041 mvcc.cc:205] Tried to move back new op lower bound from 7333603553560408064 to 7333603553482121216. Current Snapshot: MvccSnapshot[applied={T|T < 7333603553560408064}]
I20260926 13:49:15.089222 3015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "91356a680f9c4d5a9ca5eee9786f09df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91356a680f9c4d5a9ca5eee9786f09df" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40155 } } peers { permanent_uuid: "27dd313d415f45b3b75e380bb1e8bcca" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 44205 } } peers { permanent_uuid: "958ad1e870f14b83bed17afe167e71e9" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } } }
I20260926 13:49:15.089325 3015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:15.090211 2987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 958ad1e870f14b83bed17afe167e71e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "91356a680f9c4d5a9ca5eee9786f09df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91356a680f9c4d5a9ca5eee9786f09df" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 40155 } } peers { permanent_uuid: "27dd313d415f45b3b75e380bb1e8bcca" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 44205 } } peers { permanent_uuid: "958ad1e870f14b83bed17afe167e71e9" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 38213 } } }
I20260926 13:49:15.090317 2987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 958ad1e870f14b83bed17afe167e71e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:49:15.090901 3039 mvcc.cc:205] Tried to move back new op lower bound from 7333603553560408064 to 7333603553482121216. Current Snapshot: MvccSnapshot[applied={T|T < 7333603553560408064}]
I20260926 13:49:15.093979 3036 catalog_manager.cc:1347] Generated new cluster ID: 90eee3ac7d0d48d1b9be654b3f3589db
I20260926 13:49:15.094038 3036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:49:15.116398 3036 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:49:15.117556 3036 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:49:15.130235 3036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91356a680f9c4d5a9ca5eee9786f09df: Generated new TSK 0
I20260926 13:49:15.130967 3036 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 13:49:15.161432 2984 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:49:15.158717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38880 (local address 127.12.170.124:38213)
0926 13:49:15.158832 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:15.158837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:15.160481 (+ 1644us) server_negotiation.cc:407] Connection header received
0926 13:49:15.160649 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:15.160652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:15.160697 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:15.160745 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:15.161308 (+ 563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38880: BlockingRecv error: recv got EOF from 127.0.0.1:38880 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 13:49:15.162712 2834 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:49:15.159377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55110 (local address 127.12.170.126:40155)
0926 13:49:15.159480 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:15.159484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:15.161865 (+ 2381us) server_negotiation.cc:407] Connection header received
0926 13:49:15.162028 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:15.162031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:15.162079 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:15.162126 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:15.162608 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55110: BlockingRecv error: recv got EOF from 127.0.0.1:55110 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 13:49:15.163962 2911 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:49:15.159968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52600 (local address 127.12.170.125:44205)
0926 13:49:15.160070 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:15.160073 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:15.163148 (+ 3075us) server_negotiation.cc:407] Connection header received
0926 13:49:15.163294 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:15.163297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:15.163342 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:15.163389 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:15.163858 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52600: BlockingRecv error: recv got EOF from 127.0.0.1:52600 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 13:49:15.167915 2911 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:49:15.166544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52602 (local address 127.12.170.125:44205)
0926 13:49:15.166638 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:15.166642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:15.167053 (+ 411us) server_negotiation.cc:407] Connection header received
0926 13:49:15.167237 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:15.167240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:15.167303 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:15.167352 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:15.167832 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52602: BlockingRecv error: recv got EOF from 127.0.0.1:52602 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 13:49:15.172528 2911 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:49:15.171161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52616 (local address 127.12.170.125:44205)
0926 13:49:15.171251 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:15.171254 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:15.171736 (+ 482us) server_negotiation.cc:407] Connection header received
0926 13:49:15.171921 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:15.171924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:15.171968 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:15.172013 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:15.172447 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52616: BlockingRecv error: recv got EOF from 127.0.0.1:52616 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 13:49:15.177868 2911 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:49:15.176543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52618 (local address 127.12.170.125:44205)
0926 13:49:15.176631 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:15.176635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:15.177067 (+ 432us) server_negotiation.cc:407] Connection header received
0926 13:49:15.177238 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:15.177242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:15.177290 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:15.177337 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:15.177771 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52618: BlockingRecv error: recv got EOF from 127.0.0.1:52618 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 13:49:15.185266 2911 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:49:15.183887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52630 (local address 127.12.170.125:44205)
0926 13:49:15.183997 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:15.184001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:15.184431 (+ 430us) server_negotiation.cc:407] Connection header received
0926 13:49:15.184603 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:15.184606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:15.184650 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:15.184696 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:15.185178 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52630: BlockingRecv error: recv got EOF from 127.0.0.1:52630 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 13:49:15.196928 2911 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:49:15.195388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52640 (local address 127.12.170.125:44205)
0926 13:49:15.195492 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:15.195496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:15.196035 (+ 539us) server_negotiation.cc:407] Connection header received
0926 13:49:15.196220 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:15.196224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:15.196270 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:15.196318 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:15.196829 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52640: BlockingRecv error: recv got EOF from 127.0.0.1:52640 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 13:49:15.216962 2911 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:49:15.215349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52648 (local address 127.12.170.125:44205)
0926 13:49:15.215464 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:15.215469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:15.216061 (+ 592us) server_negotiation.cc:407] Connection header received
0926 13:49:15.216202 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:15.216206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:15.216263 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:15.216365 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:15.216851 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52648: BlockingRecv error: recv got EOF from 127.0.0.1:52648 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 13:49:15.252140 2911 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:49:15.250990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52660 (local address 127.12.170.125:44205)
0926 13:49:15.251131 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:15.251136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:15.251276 (+ 140us) server_negotiation.cc:407] Connection header received
0926 13:49:15.251395 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:15.251397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:15.251446 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:15.251522 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:15.252019 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52660: BlockingRecv error: recv got EOF from 127.0.0.1:52660 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 13:49:15.319113 2911 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:49:15.318079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52668 (local address 127.12.170.125:44205)
0926 13:49:15.318214 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:15.318217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:15.318325 (+ 108us) server_negotiation.cc:407] Connection header received
0926 13:49:15.318442 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:15.318444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:15.318502 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:15.318579 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:15.319015 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52668: BlockingRecv error: recv got EOF from 127.0.0.1:52668 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 13:49:15.450198 2911 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:49:15.449047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52674 (local address 127.12.170.125:44205)
0926 13:49:15.449196 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:15.449201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:15.449355 (+ 154us) server_negotiation.cc:407] Connection header received
0926 13:49:15.449503 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:15.449507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:15.449556 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:15.449623 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:15.450078 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52674: BlockingRecv error: recv got EOF from 127.0.0.1:52674 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 13:49:15.709192 2911 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:49:15.708111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52690 (local address 127.12.170.125:44205)
0926 13:49:15.708240 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:15.708244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:15.708354 (+ 110us) server_negotiation.cc:407] Connection header received
0926 13:49:15.708495 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:15.708499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:15.708546 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:15.708608 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:15.709075 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52690: BlockingRecv error: recv got EOF from 127.0.0.1:52690 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 13:49:16.005538 2998 catalog_manager.cc:1259] Loaded cluster ID: 90eee3ac7d0d48d1b9be654b3f3589db
I20260926 13:49:16.005630 2998 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 958ad1e870f14b83bed17afe167e71e9: loading cluster ID for follower catalog manager: success
I20260926 13:49:16.007336 2998 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 958ad1e870f14b83bed17afe167e71e9: acquiring CA information for follower catalog manager: success
I20260926 13:49:16.008504 2998 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 958ad1e870f14b83bed17afe167e71e9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:49:16.031844 3012 catalog_manager.cc:1259] Loaded cluster ID: 90eee3ac7d0d48d1b9be654b3f3589db
I20260926 13:49:16.031911 3012 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 27dd313d415f45b3b75e380bb1e8bcca: loading cluster ID for follower catalog manager: success
I20260926 13:49:16.033486 3012 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 27dd313d415f45b3b75e380bb1e8bcca: acquiring CA information for follower catalog manager: success
I20260926 13:49:16.034657 3012 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 27dd313d415f45b3b75e380bb1e8bcca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:49:16.226824 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:16.223054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52694 (local address 127.12.170.125:44205)
0926 13:49:16.223317 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:16.223321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:16.223385 (+ 64us) server_negotiation.cc:407] Connection header received
0926 13:49:16.223441 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:16.223444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:16.223578 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:16.223697 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:49:16.223919 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:16.224495 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:16.225433 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:16.225653 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:16.225721 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:16.225998 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:49:16.226484 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:49:16.226572 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:16.226630 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 13:49:16.226676 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":89,"threads_started":1}
I20260926 13:49:16.232949 3117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:16.229638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38882 (local address 127.12.170.124:38213)
0926 13:49:16.229855 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:16.229859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:16.229871 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:49:16.229952 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:16.229954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:16.230055 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:16.230149 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:49:16.230355 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:16.230851 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:16.231629 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:16.231897 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:16.231984 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:16.232244 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:49:16.232598 (+ 354us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:49:16.232699 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:16.232762 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 13:49:16.232815 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":55,"threads_started":1}
I20260926 13:49:16.239311 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:16.235948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52698 (local address 127.12.170.125:44205)
0926 13:49:16.236065 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:16.236069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:16.236081 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:49:16.236119 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:16.236122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:16.236162 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:16.236221 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:49:16.236534 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:16.237282 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:16.238272 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:16.238551 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:16.238635 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:16.238884 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:49:16.238955 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:49:16.238997 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:16.239144 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 13:49:16.239196 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 13:49:16.239419 3123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:16.235493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55120 (local address 127.12.170.126:40155)
0926 13:49:16.235734 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:16.235739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:16.235753 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:49:16.235837 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:16.235840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:16.235981 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:16.236322 (+ 341us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:49:16.236436 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:16.236996 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:16.238017 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:16.238339 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:16.238429 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:16.238731 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:49:16.239028 (+ 297us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:49:16.239135 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:16.239201 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 13:49:16.239263 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":59,"threads_started":1}
I20260926 13:49:16.239516 3117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:16.235543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38896 (local address 127.12.170.124:38213)
0926 13:49:16.235633 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:16.235637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:16.236200 (+ 563us) server_negotiation.cc:407] Connection header received
0926 13:49:16.236283 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:16.236285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:16.236327 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:16.236404 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:49:16.236610 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:16.237329 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:16.238310 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:16.238680 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:16.238804 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:16.239016 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:49:16.239085 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:49:16.239152 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:16.239356 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 13:49:16.239404 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 13:49:16.242311 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 2763
I20260926 13:49:16.247575 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 2838
I20260926 13:49:16.252290 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 2912
2026-09-26T13:49:16Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2897 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:49:16 dist-test-slave-hc1k krb5kdc[3128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:49:16 dist-test-slave-hc1k krb5kdc[3128](info): set up 2 sockets
Sep 26 13:49:16 dist-test-slave-hc1k krb5kdc[3128](info): commencing operation
krb5kdc: starting...
W20260926 13:49:17.199981 12969 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/krb5kdc/test-user.keytab.
Sep 26 13:49:17 dist-test-slave-hc1k krb5kdc[3128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:49:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:49:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.170.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
Entry for principal kudu/127.12.170.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab.
I20260926 13:49:17.314885 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:33151
--webserver_interface=127.12.170.126
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:44783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:33151,127.12.170.125:41597,127.12.170.124:46349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:49:17.417147 3141 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 13:49:17.417387 3141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:49:17.417408 3141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:49:17.417454 3141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:49:17.421346 3141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:49:17.421391 3141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:49:17.421409 3141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:49:17.421424 3141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:49:17.421437 3141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:49:17.426396 3141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:44783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:33151,127.12.170.125:41597,127.12.170.124:46349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.126.keytab
--principal=kudu/127.12.170.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.126:33151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.170.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:49:17.427284 3141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:49:17.428242 3141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:49:17.435835 3147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:17.435874 3146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:17.435880 3149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:49:17.435968 3141 server_base.cc:1034] running on GCE node
I20260926 13:49:17.436537 3141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:49:17.437461 3141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:49:17.438611 3141 hybrid_clock.cc:648] HybridClock initialized: now 1790430557438600 us; error 24 us; skew 500 ppm
Sep 26 13:49:17 dist-test-slave-hc1k krb5kdc[3128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430557, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:49:17.441121 3141 init.cc:377] Logged in from keytab as kudu/127.12.170.126@KRBTEST.COM (short username kudu)
I20260926 13:49:17.442039 3141 webserver.cc:460] Webserver started at http://127.12.170.126:44829/ using document root <none> and password file <none>
I20260926 13:49:17.442546 3141 fs_manager.cc:359] Metadata directory not provided
I20260926 13:49:17.442601 3141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:49:17.442782 3141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:49:17.444649 3141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-0/data/instance:
uuid: "18e65828c96845daaf849fa1f0856605"
format_stamp: "Formatted at 2026-09-26 13:49:17 on dist-test-slave-hc1k"
server_key: "e01cb96c8c0cb036f4e644a06eca8c7a"
server_key_iv: "bbdbc03c35e7c86586e4deb19fdd131a"
server_key_version: "encryptionkey@0"
I20260926 13:49:17.445171 3141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-0/wal/instance:
uuid: "18e65828c96845daaf849fa1f0856605"
format_stamp: "Formatted at 2026-09-26 13:49:17 on dist-test-slave-hc1k"
server_key: "e01cb96c8c0cb036f4e644a06eca8c7a"
server_key_iv: "bbdbc03c35e7c86586e4deb19fdd131a"
server_key_version: "encryptionkey@0"
I20260926 13:49:17.448480 3141 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 13:49:17.450647 3156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:49:17.451790 3141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:49:17.451893 3141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-0/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-0/wal
uuid: "18e65828c96845daaf849fa1f0856605"
format_stamp: "Formatted at 2026-09-26 13:49:17 on dist-test-slave-hc1k"
server_key: "e01cb96c8c0cb036f4e644a06eca8c7a"
server_key_iv: "bbdbc03c35e7c86586e4deb19fdd131a"
server_key_version: "encryptionkey@0"
I20260926 13:49:17.451980 3141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:49:17.467536 3141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:49:17.469991 3141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:49:17.470140 3141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:49:17.476506 3141 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.126:33151
I20260926 13:49:17.476781 3208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.126:33151 every 8 connection(s)
I20260926 13:49:17.477353 3141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-0/data/info.pb
I20260926 13:49:17.479382 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 3141
I20260926 13:49:17.479501 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-0/wal/instance
I20260926 13:49:17.479887 12969 external_mini_cluster.cc:1428] Setting key ca369346a6269a1cdecc6e8a44e0a650
I20260926 13:49:17.480218 3209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:49:17.484162 3209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33151 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.170.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.125@KRBTEST.COM" created.
Sep 26 13:49:17 dist-test-slave-hc1k krb5kdc[3128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430557, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:49:17.498989 3212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.486425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33151 (local address 127.0.0.1:33796)
0926 13:49:17.486914 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.486956 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 13:49:17.487891 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:49:17.488543 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:49:17.488549 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:49:17.488968 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:49:17.489200 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:17.489214 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.490670 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.490673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:17.491401 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:17.491410 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.492249 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.492252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:17.492349 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:17.492922 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:49:17.492937 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:49:17.494484 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:49:17.496371 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:17.496379 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:17.496387 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:17.496581 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:17.496855 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:17.496858 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:17.496860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:17.496957 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:17.497507 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:49:17.497513 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:49:17.497638 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 13:49:17.497775 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 13:49:17.498160 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":95,"thread_start_us":137,"threads_started":1}
I20260926 13:49:17.498989 3213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.487337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33796 (local address 127.12.170.126:33151)
0926 13:49:17.487619 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:17.487634 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:17.487706 (+ 72us) server_negotiation.cc:407] Connection header received
0926 13:49:17.488171 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:17.488176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:17.488368 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:17.488534 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:17.489344 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.490525 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.491521 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.492108 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.492281 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:17.494634 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:17.494642 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:17.494656 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:17.494684 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:17.496277 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:17.496673 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:17.496678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:17.496679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:17.496769 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:17.497035 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:17.497038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:17.497040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:17.497380 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:17.497491 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:17.497864 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 13:49:17.498036 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
I20260926 13:49:17.501252 3209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.125" port: 41597 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:49:17.501931 3212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.501683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:41597 (local address 127.0.0.1:49130)
0926 13:49:17.501792 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.501873 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.125:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 13:49:17.502133 3160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.170.125:41597: connect: Connection refused (error 111)
I20260926 13:49:17.502924 3212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.502708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:41597 (local address 127.0.0.1:49132)
0926 13:49:17.502827 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.502871 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.125:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 13:49:17.503034 3209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.125:41597: Network error: Client connection negotiation failed: client connection to 127.12.170.125:41597: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.125.keytab.
Entry for principal kudu/127.12.170.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.125.keytab.
I20260926 13:49:17.503981 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.125:41597
--webserver_interface=127.12.170.125
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.125.keytab
--principal=kudu/127.12.170.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:44783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:33151,127.12.170.125:41597,127.12.170.124:46349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:49:17.552246 3209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.125" port: 41597 } attempt: 1
I20260926 13:49:17.553095 3212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.552818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:41597 (local address 127.0.0.1:49142)
0926 13:49:17.552957 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.553021 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.125:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 13:49:17.553736 3212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.553552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:41597 (local address 127.0.0.1:49146)
0926 13:49:17.553634 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.553676 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.125:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 13:49:17.553862 3209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.125:41597: Network error: Client connection negotiation failed: client connection to 127.12.170.125:41597: connect: Connection refused (error 111)
W20260926 13:49:17.606830 3216 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 13:49:17.607057 3216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:49:17.607079 3216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:49:17.607122 3216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:49:17.610960 3216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:49:17.611003 3216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:49:17.611020 3216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:49:17.611035 3216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:49:17.611052 3216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:49:17.615947 3216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:44783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:33151,127.12.170.125:41597,127.12.170.124:46349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.125.keytab
--principal=kudu/127.12.170.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.125:41597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.170.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:49:17.616786 3216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:49:17.617652 3216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:49:17.622006 3209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.125" port: 41597 } attempt: 2
I20260926 13:49:17.622779 3212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.622507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:41597 (local address 127.0.0.1:49152)
0926 13:49:17.622638 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.622698 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.125:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 13:49:17.623363 3212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.623182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:41597 (local address 127.0.0.1:49166)
0926 13:49:17.623260 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.623307 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.125:41597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 13:49:17.623497 3209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.125:41597: Network error: Client connection negotiation failed: client connection to 127.12.170.125:41597: connect: Connection refused (error 111)
W20260926 13:49:17.625610 3222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:49:17.625675 3216 server_base.cc:1034] running on GCE node
W20260926 13:49:17.625622 3221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:17.625626 3224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:49:17.626304 3216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:49:17.627259 3216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:49:17.628418 3216 hybrid_clock.cc:648] HybridClock initialized: now 1790430557628402 us; error 25 us; skew 500 ppm
Sep 26 13:49:17 dist-test-slave-hc1k krb5kdc[3128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430557, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:49:17.630882 3216 init.cc:377] Logged in from keytab as kudu/127.12.170.125@KRBTEST.COM (short username kudu)
I20260926 13:49:17.631629 3216 webserver.cc:460] Webserver started at http://127.12.170.125:42587/ using document root <none> and password file <none>
I20260926 13:49:17.632287 3216 fs_manager.cc:359] Metadata directory not provided
I20260926 13:49:17.632370 3216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:49:17.632594 3216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:49:17.634501 3216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-1/data/instance:
uuid: "e9102e821d8e4e0794fb62493b6c08ad"
format_stamp: "Formatted at 2026-09-26 13:49:17 on dist-test-slave-hc1k"
server_key: "242f5ec420ad1d0c6e7664b4d9c6b8d4"
server_key_iv: "14ce32c8f5c3ce69a5ae8206dfa096d5"
server_key_version: "encryptionkey@0"
I20260926 13:49:17.634971 3216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-1/wal/instance:
uuid: "e9102e821d8e4e0794fb62493b6c08ad"
format_stamp: "Formatted at 2026-09-26 13:49:17 on dist-test-slave-hc1k"
server_key: "242f5ec420ad1d0c6e7664b4d9c6b8d4"
server_key_iv: "14ce32c8f5c3ce69a5ae8206dfa096d5"
server_key_version: "encryptionkey@0"
I20260926 13:49:17.638248 3216 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 13:49:17.640527 3231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:49:17.641626 3216 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:49:17.641729 3216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-1/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-1/wal
uuid: "e9102e821d8e4e0794fb62493b6c08ad"
format_stamp: "Formatted at 2026-09-26 13:49:17 on dist-test-slave-hc1k"
server_key: "242f5ec420ad1d0c6e7664b4d9c6b8d4"
server_key_iv: "14ce32c8f5c3ce69a5ae8206dfa096d5"
server_key_version: "encryptionkey@0"
I20260926 13:49:17.641829 3216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:49:17.656286 3216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:49:17.658872 3216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:49:17.659022 3216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:49:17.665189 3216 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.125:41597
I20260926 13:49:17.665431 3283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.125:41597 every 8 connection(s)
I20260926 13:49:17.666034 3216 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-1/data/info.pb
I20260926 13:49:17.668519 3284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:49:17.668742 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 3216
I20260926 13:49:17.668851 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-1/wal/instance
I20260926 13:49:17.669106 12969 external_mini_cluster.cc:1428] Setting key 0e0574ee0a873726445c4e9ef3ec92fe
I20260926 13:49:17.672413 3284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33151 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.170.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.170.124@KRBTEST.COM" created.
Sep 26 13:49:17 dist-test-slave-hc1k krb5kdc[3128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430557, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.125@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:49:17.685133 3213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.674082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33806 (local address 127.12.170.126:33151)
0926 13:49:17.674224 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:17.674229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:17.674940 (+ 711us) server_negotiation.cc:407] Connection header received
0926 13:49:17.675823 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:17.675827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:17.675883 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:17.675971 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:17.677052 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.677592 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.678806 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.679139 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.679239 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:17.681816 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:17.681819 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:17.681821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:17.681846 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:17.683112 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:17.683689 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:17.683692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:17.683694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:17.683766 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:17.684092 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:17.684094 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:17.684095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:17.684245 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:17.684340 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:17.684903 (+ 563us) server_negotiation.cc:299] Negotiation successful
0926 13:49:17.685006 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 13:49:17.686086 3287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.674374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33151 (local address 127.0.0.1:33806)
0926 13:49:17.674788 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.674836 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 13:49:17.675610 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:49:17.676102 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:49:17.676111 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:49:17.676459 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:49:17.676919 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:17.676932 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.677730 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.677733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:17.678676 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:17.678685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.679266 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.679269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:17.679422 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:17.679979 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:49:17.679998 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:49:17.681657 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:49:17.683224 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:17.683230 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:17.683244 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:17.683575 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:17.683845 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:17.683849 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:17.683851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:17.684015 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:17.684361 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:49:17.684367 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:49:17.684655 (+ 288us) client_negotiation.cc:769] Sending connection context
0926 13:49:17.684852 (+ 197us) client_negotiation.cc:240] Negotiation successful
0926 13:49:17.685166 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":104,"threads_started":1}
I20260926 13:49:17.687493 3284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.125" port: 41597 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.170.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.124.keytab.
Entry for principal kudu/127.12.170.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.124.keytab.
Sep 26 13:49:17 dist-test-slave-hc1k krb5kdc[3128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430557, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.125@KRBTEST.COM for kudu/127.12.170.125@KRBTEST.COM
I20260926 13:49:17.692854 12969 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0bSs4M/build/debug/bin/kudu
/tmp/dist-test-task0bSs4M/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.124:46349
--webserver_interface=127.12.170.124
--webserver_port=0
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.124.keytab
--principal=kudu/127.12.170.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.170.84:44783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.170.126:33151,127.12.170.125:41597,127.12.170.124:46349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:49:17.695740 3287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.687962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:41597 (local address 127.0.0.1:49176)
0926 13:49:17.688109 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.688121 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:49:17.688179 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:49:17.688594 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:49:17.688596 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:49:17.688720 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:49:17.688820 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:17.688825 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.689794 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.689798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:17.690369 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:17.690378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.690907 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.690910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:17.690981 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:17.691320 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:49:17.691331 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:49:17.692600 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:49:17.694347 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:17.694349 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:17.694351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:17.694518 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:17.694818 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:17.694821 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:17.694822 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:17.694909 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:17.695345 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:49:17.695347 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:49:17.695416 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 13:49:17.695495 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 13:49:17.695587 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260926 13:49:17.695906 3289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.688043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49176 (local address 127.12.170.125:41597)
0926 13:49:17.688295 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:17.688304 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:17.688321 (+ 17us) server_negotiation.cc:407] Connection header received
0926 13:49:17.688354 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:17.688358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:17.688489 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:17.688585 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:17.688932 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.689685 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.690473 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.690818 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.690894 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:17.692720 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:17.692727 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:17.692737 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:17.692756 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:17.694228 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:17.694612 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:17.694617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:17.694619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:17.694706 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:17.694998 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:17.695001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:17.695003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:17.695248 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:17.695373 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:17.695576 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 13:49:17.695725 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":91,"threads_started":1}
I20260926 13:49:17.697584 3284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:49:17.698170 3287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.697953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:46349 (local address 127.0.0.1:41470)
0926 13:49:17.698052 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.698119 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:46349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 13:49:17.698365 3235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.170.124:46349: connect: Connection refused (error 111)
I20260926 13:49:17.699129 3287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.698941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:46349 (local address 127.0.0.1:41474)
0926 13:49:17.699033 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.699082 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:46349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 13:49:17.699287 3284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.124:46349: Network error: Client connection negotiation failed: client connection to 127.12.170.124:46349: connect: Connection refused (error 111)
I20260926 13:49:17.714661 3209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.125" port: 41597 } attempt: 3
Sep 26 13:49:17 dist-test-slave-hc1k krb5kdc[3128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430557, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for kudu/127.12.170.125@KRBTEST.COM
I20260926 13:49:17.723315 3212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.715066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:41597 (local address 127.0.0.1:49186)
0926 13:49:17.715190 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.715206 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:49:17.715314 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:49:17.715541 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:49:17.715544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:49:17.715756 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:49:17.716185 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:17.716193 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.716970 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.716974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:17.717526 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:17.717535 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.718388 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.718392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:17.718501 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:17.718957 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:49:17.718974 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:49:17.720514 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:49:17.722142 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:17.722145 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:17.722146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:17.722321 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:17.722566 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:17.722568 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:17.722569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:17.722629 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:17.722971 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:49:17.722972 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:49:17.723044 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 13:49:17.723113 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 13:49:17.723203 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 13:49:17.723407 3289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.715112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49186 (local address 127.12.170.125:41597)
0926 13:49:17.715208 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:17.715212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:17.715254 (+ 42us) server_negotiation.cc:407] Connection header received
0926 13:49:17.715406 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:17.715409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:17.715448 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:17.715512 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:17.716350 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.716863 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.717637 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.718271 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.718374 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:17.720646 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:17.720649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:17.720652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:17.720676 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:17.722044 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:17.722404 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:17.722407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:17.722409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:17.722491 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:17.722698 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:17.722701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:17.722702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:17.722889 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:17.722988 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:17.723174 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 13:49:17.723274 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 13:49:17.724035 3209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:49:17.724610 3212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.724416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:46349 (local address 127.0.0.1:41484)
0926 13:49:17.724514 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.724557 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:46349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 13:49:17.725196 3212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.724984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:46349 (local address 127.0.0.1:41500)
0926 13:49:17.725113 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.725153 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:46349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260926 13:49:17.725329 3209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.124:46349: Network error: Client connection negotiation failed: client connection to 127.12.170.124:46349: connect: Connection refused (error 111)
I20260926 13:49:17.748502 3284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } attempt: 1
I20260926 13:49:17.749133 3287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.748902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:46349 (local address 127.0.0.1:41514)
0926 13:49:17.749023 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.749073 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:46349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 13:49:17.749713 3287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.749535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:46349 (local address 127.0.0.1:41526)
0926 13:49:17.749625 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.749665 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:46349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 13:49:17.749840 3284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.124:46349: Network error: Client connection negotiation failed: client connection to 127.12.170.124:46349: connect: Connection refused (error 111)
I20260926 13:49:17.756466 3209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } attempt: 1
I20260926 13:49:17.757016 3212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.756819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:46349 (local address 127.0.0.1:41538)
0926 13:49:17.756920 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.756963 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:46349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 13:49:17.757598 3212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.757412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:46349 (local address 127.0.0.1:41544)
0926 13:49:17.757506 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.757544 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:46349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 13:49:17.757704 3209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.124:46349: Network error: Client connection negotiation failed: client connection to 127.12.170.124:46349: connect: Connection refused (error 111)
W20260926 13:49:17.797817 3290 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 13:49:17.798050 3290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:49:17.798071 3290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:49:17.798122 3290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:49:17.802124 3290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:49:17.802174 3290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:49:17.802192 3290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:49:17.802206 3290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:49:17.802220 3290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:49:17.807237 3290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.170.84:44783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.170.126:33151,127.12.170.125:41597,127.12.170.124:46349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/krb5kdc/kudu_127.12.170.124.keytab
--principal=kudu/127.12.170.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.170.124:46349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.170.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 12:47:21 UTC on ccycloud.quasar-ftxjdu.root.comops.site
build id 4673841
I20260926 13:49:17.808190 3290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:49:17.809103 3290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:49:17.816793 3290 server_base.cc:1034] running on GCE node
W20260926 13:49:17.816740 3299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:17.816748 3296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:49:17.816754 3297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:49:17.817432 3290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:49:17.817972 3284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } attempt: 2
I20260926 13:49:17.818485 3290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:49:17.818688 3287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.818398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:46349 (local address 127.0.0.1:41560)
0926 13:49:17.818537 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.818600 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:46349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 13:49:17.819294 3287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.819115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:46349 (local address 127.0.0.1:41566)
0926 13:49:17.819197 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.819239 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:46349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 13:49:17.819406 3284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.124:46349: Network error: Client connection negotiation failed: client connection to 127.12.170.124:46349: connect: Connection refused (error 111)
I20260926 13:49:17.819638 3290 hybrid_clock.cc:648] HybridClock initialized: now 1790430557819540 us; error 105 us; skew 500 ppm
Sep 26 13:49:17 dist-test-slave-hc1k krb5kdc[3128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430557, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:49:17.822012 3290 init.cc:377] Logged in from keytab as kudu/127.12.170.124@KRBTEST.COM (short username kudu)
I20260926 13:49:17.822793 3290 webserver.cc:460] Webserver started at http://127.12.170.124:45677/ using document root <none> and password file <none>
I20260926 13:49:17.823266 3290 fs_manager.cc:359] Metadata directory not provided
I20260926 13:49:17.823324 3290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:49:17.823503 3290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:49:17.825400 3290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-2/data/instance:
uuid: "582f42ac3cb54b83aea0d1ca6205ac0e"
format_stamp: "Formatted at 2026-09-26 13:49:17 on dist-test-slave-hc1k"
server_key: "d318e894a3d205f11bbc3b1e3312e47f"
server_key_iv: "1ba25a2a18c7464805c8321ec2ff5d4a"
server_key_version: "encryptionkey@0"
I20260926 13:49:17.825896 3290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-2/wal/instance:
uuid: "582f42ac3cb54b83aea0d1ca6205ac0e"
format_stamp: "Formatted at 2026-09-26 13:49:17 on dist-test-slave-hc1k"
server_key: "d318e894a3d205f11bbc3b1e3312e47f"
server_key_iv: "1ba25a2a18c7464805c8321ec2ff5d4a"
server_key_version: "encryptionkey@0"
I20260926 13:49:17.829208 3290 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 13:49:17.831303 3306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:49:17.832464 3290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:49:17.832573 3290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-2/data,/tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-2/wal
uuid: "582f42ac3cb54b83aea0d1ca6205ac0e"
format_stamp: "Formatted at 2026-09-26 13:49:17 on dist-test-slave-hc1k"
server_key: "d318e894a3d205f11bbc3b1e3312e47f"
server_key_iv: "1ba25a2a18c7464805c8321ec2ff5d4a"
server_key_version: "encryptionkey@0"
I20260926 13:49:17.832692 3290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:49:17.832841 3209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } attempt: 2
I20260926 13:49:17.833448 3212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.833212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:46349 (local address 127.0.0.1:41578)
0926 13:49:17.833327 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.833377 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:46349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 13:49:17.834105 3212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.833927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:46349 (local address 127.0.0.1:41582)
0926 13:49:17.834012 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.834051 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.170.124:46349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 13:49:17.834216 3209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.170.124:46349: Network error: Client connection negotiation failed: client connection to 127.12.170.124:46349: connect: Connection refused (error 111)
I20260926 13:49:17.844942 3290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:49:17.847913 3290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:49:17.848081 3290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:49:17.854705 3290 rpc_server.cc:307] RPC server started. Bound to: 127.12.170.124:46349
I20260926 13:49:17.854916 3358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.170.124:46349 every 8 connection(s)
I20260926 13:49:17.855677 3290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-2/data/info.pb
I20260926 13:49:17.858245 12969 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0bSs4M/build/debug/bin/kudu as pid 3290
I20260926 13:49:17.858275 3359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:49:17.858383 12969 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0bSs4M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790430291939677-12969-0/minicluster-data/master-2/wal/instance
I20260926 13:49:17.858686 12969 external_mini_cluster.cc:1428] Setting key f932c2be89f82fdb319611341938ce55
I20260926 13:49:17.862676 3359 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33151 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:49:17 dist-test-slave-hc1k krb5kdc[3128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.124@KRBTEST.COM
Sep 26 13:49:17 dist-test-slave-hc1k krb5kdc[3128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430557, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.124@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:49:17.872743 3362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.859917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.12.170.124:46349)
0926 13:49:17.860320 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:17.860335 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:17.860370 (+ 35us) server_negotiation.cc:407] Connection header received
0926 13:49:17.861297 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:17.861316 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:17.861638 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:17.861914 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:17.862741 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.863804 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.864810 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.865364 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.865485 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:17.868128 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:17.868135 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:17.868148 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:17.868178 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:17.869895 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:17.870261 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:17.870268 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:17.870274 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:17.870392 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:17.870642 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:17.870645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:17.870647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:17.871148 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:17.871339 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:17.871592 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 13:49:17.871875 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":106,"threads_started":1}
I20260926 13:49:17.873246 3363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.864166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.126:33151 (local address 127.0.0.1:33818)
0926 13:49:17.864540 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.864586 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 13:49:17.864701 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:49:17.864993 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:49:17.864997 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:49:17.865295 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:49:17.865425 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:17.865431 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.866297 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.866301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:17.867153 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:17.867162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.867806 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.867811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:17.867914 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:17.868411 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:49:17.868422 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:49:17.869887 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:49:17.871642 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:17.871669 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:17.871674 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:17.871931 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:17.872210 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:17.872212 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:17.872213 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:17.872276 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:17.872707 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:49:17.872712 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:49:17.872803 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 13:49:17.872987 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 13:49:17.873120 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":127,"threads_started":1}
I20260926 13:49:17.873344 3213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.864220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33818 (local address 127.12.170.126:33151)
0926 13:49:17.864394 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:17.864399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:17.864668 (+ 269us) server_negotiation.cc:407] Connection header received
0926 13:49:17.864813 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:17.864816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:17.864871 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:17.864956 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:17.865548 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.866174 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.867272 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.867610 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.867761 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:17.870017 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:17.870022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:17.870025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:17.870058 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:17.871530 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:17.872023 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:17.872027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:17.872029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:17.872121 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:17.872347 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:17.872351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:17.872353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:17.872581 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:17.872688 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:17.873059 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 13:49:17.873176 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 13:49:17.874203 3359 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.125" port: 41597 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:49:17 dist-test-slave-hc1k krb5kdc[3128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430557, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.124@KRBTEST.COM for kudu/127.12.170.125@KRBTEST.COM
Sep 26 13:49:17 dist-test-slave-hc1k krb5kdc[3128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.126@KRBTEST.COM
I20260926 13:49:17.881424 3363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.874592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.125:41597 (local address 127.0.0.1:49188)
0926 13:49:17.874714 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.874724 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:49:17.874783 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:49:17.875030 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:49:17.875032 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:49:17.875192 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:49:17.875306 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:17.875314 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.876029 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.876033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:17.876542 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:17.876550 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.876967 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.876972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:17.877076 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:17.877502 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:49:17.877514 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:49:17.878513 (+ 999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:49:17.880100 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:17.880103 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:17.880105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:17.880320 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:17.880615 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:17.880618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:17.880619 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:17.880688 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:17.881099 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:49:17.881102 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:49:17.881169 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 13:49:17.881243 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 13:49:17.881327 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 13:49:17.881605 3289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.874709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49188 (local address 127.12.170.125:41597)
0926 13:49:17.874837 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:17.874841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:17.874855 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:49:17.874893 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:17.874895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:17.874931 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:17.875011 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:17.875417 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.875929 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.876661 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.876869 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.876940 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:17.878629 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:17.878632 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:17.878634 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:17.878655 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:17.880006 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:17.880417 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:17.880421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:17.880423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:17.880521 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:17.880778 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:17.880781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:17.880782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:17.881006 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:17.881118 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:17.881333 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 13:49:17.881443 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 13:49:17.882155 3359 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:49:17.883237 3213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.875012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33820 (local address 127.12.170.126:33151)
0926 13:49:17.875133 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:17.875137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:17.875216 (+ 79us) server_negotiation.cc:407] Connection header received
0926 13:49:17.875296 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:17.875299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:17.875341 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:17.875413 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:17.876062 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.876525 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.877410 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.877615 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.877704 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:17.879987 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:17.879991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:17.879993 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:17.880019 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:17.881546 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:17.881987 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:17.881991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:17.881993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:17.882074 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:17.882333 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:17.882336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:17.882338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:17.882497 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:17.882674 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:17.883060 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 13:49:17.883144 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 13:49:17 dist-test-slave-hc1k krb5kdc[3128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430557, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.124@KRBTEST.COM for kudu/127.12.170.124@KRBTEST.COM
Sep 26 13:49:17 dist-test-slave-hc1k krb5kdc[3128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.170.125@KRBTEST.COM
I20260926 13:49:17.889966 3363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.882535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:46349 (local address 127.0.0.1:41602)
0926 13:49:17.882690 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.882704 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:49:17.882775 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:49:17.882995 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:49:17.882998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:49:17.883154 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:49:17.883242 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:17.883249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.884440 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.884442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:17.884906 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:17.884913 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.885432 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.885436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:17.885514 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:17.885869 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:49:17.885883 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:49:17.887101 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:49:17.888633 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:17.888636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:17.888637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:17.888838 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:17.889082 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:17.889086 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:17.889088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:17.889178 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:17.889541 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:49:17.889544 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:49:17.889619 (+ 75us) client_negotiation.cc:769] Sending connection context
0926 13:49:17.889701 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 13:49:17.889815 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260926 13:49:17.889989 3362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.882634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41602 (local address 127.12.170.124:46349)
0926 13:49:17.882718 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:17.882722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:17.882738 (+ 16us) server_negotiation.cc:407] Connection header received
0926 13:49:17.882872 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:17.882874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:17.882908 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:17.882959 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:17.883372 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.884358 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.885010 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.885338 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.885415 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:17.887227 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:17.887231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:17.887233 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:17.887254 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:17.888532 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:17.888935 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:17.888938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:17.888939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:17.889005 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:17.889267 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:17.889270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:17.889271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:17.889466 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:17.889552 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:17.889748 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 13:49:17.889867 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":37}
I20260926 13:49:17.892629 3289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.884226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49194 (local address 127.12.170.125:41597)
0926 13:49:17.884316 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:17.884320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:17.884382 (+ 62us) server_negotiation.cc:407] Connection header received
0926 13:49:17.884506 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:17.884510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:17.884553 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:17.884632 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:17.885236 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.885824 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.886640 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.886860 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.886946 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:17.889281 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:17.889285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:17.889287 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:17.889308 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:17.891121 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:17.891500 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:17.891503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:17.891505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:17.891580 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:17.891863 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:17.891866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:17.891868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:17.892031 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:17.892251 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:17.892381 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 13:49:17.892481 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 13:49:17.892619 3359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 582f42ac3cb54b83aea0d1ca6205ac0e: Bootstrap starting.
I20260926 13:49:17.895044 3359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 582f42ac3cb54b83aea0d1ca6205ac0e: Neither blocks nor log segments found. Creating new log.
I20260926 13:49:17.895480 12969 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:49:17.895862 3359 log.cc:824] T 00000000000000000000000000000000 P 582f42ac3cb54b83aea0d1ca6205ac0e: Log is configured to *not* fsync() on all Append() calls
I20260926 13:49:17.896634 3183 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "18e65828c96845daaf849fa1f0856605"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33820
I20260926 13:49:17.897632 3359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 582f42ac3cb54b83aea0d1ca6205ac0e: No bootstrap required, opened a new log
I20260926 13:49:17.900525 3359 raft_consensus.cc:348] T 00000000000000000000000000000000 P 582f42ac3cb54b83aea0d1ca6205ac0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e65828c96845daaf849fa1f0856605" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33151 } } peers { permanent_uuid: "e9102e821d8e4e0794fb62493b6c08ad" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 41597 } } peers { permanent_uuid: "582f42ac3cb54b83aea0d1ca6205ac0e" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } }
I20260926 13:49:17.900722 3359 raft_consensus.cc:729] T 00000000000000000000000000000000 P 582f42ac3cb54b83aea0d1ca6205ac0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 582f42ac3cb54b83aea0d1ca6205ac0e, State: Initialized, Role: FOLLOWER
I20260926 13:49:17.901018 3359 consensus_queue.cc:260] T 00000000000000000000000000000000 P 582f42ac3cb54b83aea0d1ca6205ac0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e65828c96845daaf849fa1f0856605" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33151 } } peers { permanent_uuid: "e9102e821d8e4e0794fb62493b6c08ad" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 41597 } } peers { permanent_uuid: "582f42ac3cb54b83aea0d1ca6205ac0e" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } }
I20260926 13:49:17.901439 3365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 582f42ac3cb54b83aea0d1ca6205ac0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e65828c96845daaf849fa1f0856605" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33151 } } peers { permanent_uuid: "e9102e821d8e4e0794fb62493b6c08ad" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 41597 } } peers { permanent_uuid: "582f42ac3cb54b83aea0d1ca6205ac0e" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } } }
I20260926 13:49:17.901538 3365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 582f42ac3cb54b83aea0d1ca6205ac0e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:49:17.902277 3359 sys_catalog.cc:564] T 00000000000000000000000000000000 P 582f42ac3cb54b83aea0d1ca6205ac0e [sys.catalog]: configured and running, proceeding with master startup.
W20260926 13:49:17.906875 3376 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 582f42ac3cb54b83aea0d1ca6205ac0e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:49:17.906945 3376 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:49:17.905822 3290 master_runner.cc:186] Error getting master registration for 127.12.170.126:33151: OK, instance_id { permanent_uuid: "18e65828c96845daaf849fa1f0856605" instance_seqno: 1790430557474732 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 13:49:17.908699 3290 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 13:49:17.910550 3284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } attempt: 3
Sep 26 13:49:17 dist-test-slave-hc1k krb5kdc[3128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430557, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.125@KRBTEST.COM for kudu/127.12.170.124@KRBTEST.COM
I20260926 13:49:17.918921 3362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.911039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41614 (local address 127.12.170.124:46349)
0926 13:49:17.911137 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:17.911141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:17.911153 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:49:17.911229 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:17.911232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:17.911271 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:17.911338 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:17.911877 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.912626 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.913651 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.913979 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.914060 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:17.916308 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:17.916312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:17.916314 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:17.916338 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:17.917539 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:17.917902 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:17.917906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:17.917907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:17.917972 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:17.918199 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:17.918202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:17.918204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:17.918378 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:17.918460 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:17.918691 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 13:49:17.918783 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 13:49:17.918922 3287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.910956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:46349 (local address 127.0.0.1:41614)
0926 13:49:17.911067 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.911081 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:49:17.911159 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:49:17.911390 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:49:17.911394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:49:17.911597 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:49:17.911760 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:17.911769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.912733 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.912737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:17.913538 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:17.913550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.914080 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.914085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:17.914176 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:17.914687 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:49:17.914704 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:49:17.916192 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:49:17.917635 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:17.917638 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:17.917640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:17.917820 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:17.918049 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:17.918052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:17.918054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:17.918127 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:17.918464 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:49:17.918467 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:49:17.918554 (+ 87us) client_negotiation.cc:769] Sending connection context
0926 13:49:17.918636 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 13:49:17.918760 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 13:49:17.922144 3284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9102e821d8e4e0794fb62493b6c08ad: Bootstrap starting.
I20260926 13:49:17.924800 3284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9102e821d8e4e0794fb62493b6c08ad: Neither blocks nor log segments found. Creating new log.
I20260926 13:49:17.925629 3284 log.cc:824] T 00000000000000000000000000000000 P e9102e821d8e4e0794fb62493b6c08ad: Log is configured to *not* fsync() on all Append() calls
I20260926 13:49:17.927615 3284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9102e821d8e4e0794fb62493b6c08ad: No bootstrap required, opened a new log
I20260926 13:49:17.931448 3284 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9102e821d8e4e0794fb62493b6c08ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e65828c96845daaf849fa1f0856605" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33151 } } peers { permanent_uuid: "e9102e821d8e4e0794fb62493b6c08ad" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 41597 } } peers { permanent_uuid: "582f42ac3cb54b83aea0d1ca6205ac0e" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } }
I20260926 13:49:17.931686 3284 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9102e821d8e4e0794fb62493b6c08ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9102e821d8e4e0794fb62493b6c08ad, State: Initialized, Role: FOLLOWER
I20260926 13:49:17.932003 3284 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9102e821d8e4e0794fb62493b6c08ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e65828c96845daaf849fa1f0856605" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33151 } } peers { permanent_uuid: "e9102e821d8e4e0794fb62493b6c08ad" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 41597 } } peers { permanent_uuid: "582f42ac3cb54b83aea0d1ca6205ac0e" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } }
I20260926 13:49:17.932451 3379 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9102e821d8e4e0794fb62493b6c08ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e65828c96845daaf849fa1f0856605" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33151 } } peers { permanent_uuid: "e9102e821d8e4e0794fb62493b6c08ad" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 41597 } } peers { permanent_uuid: "582f42ac3cb54b83aea0d1ca6205ac0e" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } } }
I20260926 13:49:17.932588 3379 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9102e821d8e4e0794fb62493b6c08ad [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:49:17.933260 3284 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9102e821d8e4e0794fb62493b6c08ad [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:49:17.933320 3209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } attempt: 3
Sep 26 13:49:17 dist-test-slave-hc1k krb5kdc[3128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790430557, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.170.126@KRBTEST.COM for kudu/127.12.170.124@KRBTEST.COM
W20260926 13:49:17.939096 3390 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e9102e821d8e4e0794fb62493b6c08ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:49:17.939204 3390 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:49:17.936885 3216 master_runner.cc:186] Error getting master registration for 127.12.170.126:33151: OK, instance_id { permanent_uuid: "18e65828c96845daaf849fa1f0856605" instance_seqno: 1790430557474732 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 13:49:17.940016 3216 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 13:49:17.942467 3212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.933745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.170.124:46349 (local address 127.0.0.1:41624)
0926 13:49:17.933876 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 13:49:17.933890 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:49:17.933973 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:49:17.934207 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:49:17.934209 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:49:17.934395 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:49:17.934493 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:17.934500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.935474 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.935491 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:17.936285 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:49:17.936298 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.936963 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.936967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:49:17.937052 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:17.937609 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:49:17.937623 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:49:17.939057 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:49:17.940952 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:17.940955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:17.940958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:17.941184 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:17.941453 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:49:17.941456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:49:17.941458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:49:17.941565 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:49:17.941955 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:49:17.941958 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:49:17.942071 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 13:49:17.942170 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 13:49:17.942306 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 13:49:17.942493 3362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.933926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41624 (local address 127.12.170.124:46349)
0926 13:49:17.934034 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:17.934038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:17.934050 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:49:17.934081 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:17.934084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:17.934119 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:17.934179 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:17.934619 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.935366 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.936434 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.936728 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.936817 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:17.939231 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:17.939235 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:17.939238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:17.939263 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:17.940856 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:17.941289 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:17.941292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:17.941294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:17.941374 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:17.941656 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:17.941659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:17.941661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:17.941871 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:17.941962 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:17.942245 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 13:49:17.942356 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 13:49:17.945581 3209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605: Bootstrap starting.
I20260926 13:49:17.951042 3209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605: Neither blocks nor log segments found. Creating new log.
I20260926 13:49:17.951920 3209 log.cc:824] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605: Log is configured to *not* fsync() on all Append() calls
I20260926 13:49:17.953877 3209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605: No bootstrap required, opened a new log
I20260926 13:49:17.956847 3209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e65828c96845daaf849fa1f0856605" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33151 } } peers { permanent_uuid: "e9102e821d8e4e0794fb62493b6c08ad" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 41597 } } peers { permanent_uuid: "582f42ac3cb54b83aea0d1ca6205ac0e" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } }
I20260926 13:49:17.957055 3209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18e65828c96845daaf849fa1f0856605, State: Initialized, Role: FOLLOWER
I20260926 13:49:17.957367 3209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e65828c96845daaf849fa1f0856605" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33151 } } peers { permanent_uuid: "e9102e821d8e4e0794fb62493b6c08ad" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 41597 } } peers { permanent_uuid: "582f42ac3cb54b83aea0d1ca6205ac0e" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } }
I20260926 13:49:17.958559 3209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:49:17.958614 3183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 13:49:17.958721 3183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:49:17.959853 3183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e65828c96845daaf849fa1f0856605" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33151 } } peers { permanent_uuid: "e9102e821d8e4e0794fb62493b6c08ad" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 41597 } } peers { permanent_uuid: "582f42ac3cb54b83aea0d1ca6205ac0e" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } }
W20260926 13:49:17.963243 3404 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:49:17.963342 3404 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:49:17.965404 3258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "18e65828c96845daaf849fa1f0856605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e9102e821d8e4e0794fb62493b6c08ad"
I20260926 13:49:17.965600 3258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9102e821d8e4e0794fb62493b6c08ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:49:17.966733 3141 master_runner.cc:263] No leader master found from master 18e65828c96845daaf849fa1f0856605
I20260926 13:49:17.966799 3141 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 13:49:17.966806 3258 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e9102e821d8e4e0794fb62493b6c08ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18e65828c96845daaf849fa1f0856605 in term 1.
I20260926 13:49:17.967267 3160 leader_election.cc:302] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18e65828c96845daaf849fa1f0856605, e9102e821d8e4e0794fb62493b6c08ad; no voters:
I20260926 13:49:17.968107 3333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "18e65828c96845daaf849fa1f0856605" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "582f42ac3cb54b83aea0d1ca6205ac0e"
I20260926 13:49:17.968312 3333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 582f42ac3cb54b83aea0d1ca6205ac0e [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:49:17.969485 3333 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 582f42ac3cb54b83aea0d1ca6205ac0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18e65828c96845daaf849fa1f0856605 in term 1.
I20260926 13:49:17.969929 3183 leader_election.cc:288] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605 [CANDIDATE]: Term 1 election: Requested vote from peers e9102e821d8e4e0794fb62493b6c08ad (127.12.170.125:41597), 582f42ac3cb54b83aea0d1ca6205ac0e (127.12.170.124:46349)
W20260926 13:49:17.971451 3162 tablet.cc:2369] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:49:17.971776 3393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e65828c96845daaf849fa1f0856605" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33151 } } peers { permanent_uuid: "e9102e821d8e4e0794fb62493b6c08ad" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 41597 } } peers { permanent_uuid: "582f42ac3cb54b83aea0d1ca6205ac0e" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } } }
I20260926 13:49:17.971899 3393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:49:17.971972 3393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:49:17.972041 3393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605 [term 1 LEADER]: Becoming Leader. State: Replica: 18e65828c96845daaf849fa1f0856605, State: Running, Role: LEADER
I20260926 13:49:17.972333 3393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e65828c96845daaf849fa1f0856605" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33151 } } peers { permanent_uuid: "e9102e821d8e4e0794fb62493b6c08ad" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 41597 } } peers { permanent_uuid: "582f42ac3cb54b83aea0d1ca6205ac0e" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } }
I20260926 13:49:17.973979 3393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18e65828c96845daaf849fa1f0856605. Latest consensus state: current_term: 1 leader_uuid: "18e65828c96845daaf849fa1f0856605" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e65828c96845daaf849fa1f0856605" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33151 } } peers { permanent_uuid: "e9102e821d8e4e0794fb62493b6c08ad" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 41597 } } peers { permanent_uuid: "582f42ac3cb54b83aea0d1ca6205ac0e" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } } }
I20260926 13:49:17.974088 3393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605 [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:17.974431 3412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:49:17.975287 3412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:49:17.980063 3333 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 582f42ac3cb54b83aea0d1ca6205ac0e [term 1 FOLLOWER]: Refusing update from remote peer 18e65828c96845daaf849fa1f0856605: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:49:17.980744 3406 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582f42ac3cb54b83aea0d1ca6205ac0e" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:49:17.984514 3258 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e9102e821d8e4e0794fb62493b6c08ad [term 1 FOLLOWER]: Refusing update from remote peer 18e65828c96845daaf849fa1f0856605: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:49:17.984798 3365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 582f42ac3cb54b83aea0d1ca6205ac0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18e65828c96845daaf849fa1f0856605. Latest consensus state: current_term: 1 leader_uuid: "18e65828c96845daaf849fa1f0856605" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e65828c96845daaf849fa1f0856605" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33151 } } peers { permanent_uuid: "e9102e821d8e4e0794fb62493b6c08ad" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 41597 } } peers { permanent_uuid: "582f42ac3cb54b83aea0d1ca6205ac0e" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } } }
I20260926 13:49:17.984916 3365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 582f42ac3cb54b83aea0d1ca6205ac0e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:49:17.985071 3393 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9102e821d8e4e0794fb62493b6c08ad" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 41597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:49:17.989071 3379 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9102e821d8e4e0794fb62493b6c08ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18e65828c96845daaf849fa1f0856605. Latest consensus state: current_term: 1 leader_uuid: "18e65828c96845daaf849fa1f0856605" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e65828c96845daaf849fa1f0856605" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33151 } } peers { permanent_uuid: "e9102e821d8e4e0794fb62493b6c08ad" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 41597 } } peers { permanent_uuid: "582f42ac3cb54b83aea0d1ca6205ac0e" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } } }
I20260926 13:49:17.989199 3379 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9102e821d8e4e0794fb62493b6c08ad [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:49:17.994012 3213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.976383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33822 (local address 127.12.170.126:33151)
0926 13:49:17.976475 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:17.976479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:17.979665 (+ 3186us) server_negotiation.cc:407] Connection header received
0926 13:49:17.979845 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:17.979848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:17.979893 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:17.979937 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:17.980886 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.981663 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.982793 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.983134 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:17.983200 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:17.990409 (+ 7209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:17.990413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:17.990416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:17.990442 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:17.992236 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:17.992712 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:17.992715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:17.992717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:17.992811 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:17.993113 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:17.993117 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:17.993119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:17.993333 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:17.993428 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:17.993756 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 13:49:17.993866 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 13:49:17.995366 3393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "18e65828c96845daaf849fa1f0856605" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e65828c96845daaf849fa1f0856605" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33151 } } peers { permanent_uuid: "e9102e821d8e4e0794fb62493b6c08ad" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 41597 } } peers { permanent_uuid: "582f42ac3cb54b83aea0d1ca6205ac0e" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } } }
I20260926 13:49:17.995462 3393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605 [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:17.996450 3365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 582f42ac3cb54b83aea0d1ca6205ac0e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "18e65828c96845daaf849fa1f0856605" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e65828c96845daaf849fa1f0856605" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33151 } } peers { permanent_uuid: "e9102e821d8e4e0794fb62493b6c08ad" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 41597 } } peers { permanent_uuid: "582f42ac3cb54b83aea0d1ca6205ac0e" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } } }
I20260926 13:49:17.996551 3365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 582f42ac3cb54b83aea0d1ca6205ac0e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:49:17.997733 3415 mvcc.cc:205] Tried to move back new op lower bound from 7333603565480706048 to 7333603565457850368. Current Snapshot: MvccSnapshot[applied={T|T < 7333603565480706048}]
I20260926 13:49:17.999157 3417 mvcc.cc:205] Tried to move back new op lower bound from 7333603565480706048 to 7333603565457850368. Current Snapshot: MvccSnapshot[applied={T|T < 7333603565480706048}]
I20260926 13:49:18.002497 3412 catalog_manager.cc:1347] Generated new cluster ID: 19338c88d84b4cc49e63f1758ccf0f2b
I20260926 13:49:18.002550 3412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:49:18.003070 3406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "18e65828c96845daaf849fa1f0856605" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e65828c96845daaf849fa1f0856605" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33151 } } peers { permanent_uuid: "e9102e821d8e4e0794fb62493b6c08ad" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 41597 } } peers { permanent_uuid: "582f42ac3cb54b83aea0d1ca6205ac0e" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } } }
I20260926 13:49:18.003166 3406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605 [sys.catalog]: This master's current role is: LEADER
I20260926 13:49:18.004112 3379 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9102e821d8e4e0794fb62493b6c08ad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "18e65828c96845daaf849fa1f0856605" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18e65828c96845daaf849fa1f0856605" member_type: VOTER last_known_addr { host: "127.12.170.126" port: 33151 } } peers { permanent_uuid: "e9102e821d8e4e0794fb62493b6c08ad" member_type: VOTER last_known_addr { host: "127.12.170.125" port: 41597 } } peers { permanent_uuid: "582f42ac3cb54b83aea0d1ca6205ac0e" member_type: VOTER last_known_addr { host: "127.12.170.124" port: 46349 } } }
I20260926 13:49:18.004213 3379 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9102e821d8e4e0794fb62493b6c08ad [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:49:18.008538 3419 mvcc.cc:205] Tried to move back new op lower bound from 7333603565480706048 to 7333603565457850368. Current Snapshot: MvccSnapshot[applied={T|T < 7333603565480706048}]
I20260926 13:49:18.015796 3362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.976983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41632 (local address 127.12.170.124:46349)
0926 13:49:17.977081 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:17.977085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:17.994546 (+ 17461us) server_negotiation.cc:407] Connection header received
0926 13:49:17.995366 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:17.995369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:17.995416 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:17.995464 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:17.999250 (+ 3786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:18.000017 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:18.005426 (+ 5409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:18.005755 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:18.005832 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:18.008731 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:18.008735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:18.008738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:18.008765 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:18.011331 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:18.013600 (+ 2269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:18.013603 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:18.013606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:18.013686 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:18.014320 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:18.014324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:18.014326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:18.014524 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:18.014624 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:18.015501 (+ 877us) server_negotiation.cc:299] Negotiation successful
0926 13:49:18.015618 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 13:49:18.017367 3289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:17.976026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49198 (local address 127.12.170.125:41597)
0926 13:49:17.976145 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:17.976150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:17.990908 (+ 14758us) server_negotiation.cc:407] Connection header received
0926 13:49:17.991434 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:17.991437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:17.991484 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:17.991529 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:17.997405 (+ 5876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:17.998130 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:18.004859 (+ 6729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:18.005215 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:18.005295 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:18.010429 (+ 5134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:49:18.010433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:49:18.010436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:49:18.010461 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:49:18.012506 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:18.013966 (+ 1460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:18.013969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:18.013971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:18.014044 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:49:18.014888 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:49:18.014892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:49:18.014893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:49:18.015085 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:49:18.015190 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:18.017113 (+ 1923us) server_negotiation.cc:299] Negotiation successful
0926 13:49:18.017219 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 13:49:18.036886 3412 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:49:18.038199 3412 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:49:18.055634 3412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18e65828c96845daaf849fa1f0856605: Generated new TSK 0
I20260926 13:49:18.056587 3412 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 13:49:18.079293 3289 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:49:18.077019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49208 (local address 127.12.170.125:41597)
0926 13:49:18.077132 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:18.077136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:18.078239 (+ 1103us) server_negotiation.cc:407] Connection header received
0926 13:49:18.078437 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:18.078440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:18.078489 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:18.078537 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:18.079151 (+ 614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49208: BlockingRecv error: recv got EOF from 127.0.0.1:49208 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 13:49:18.080525 3362 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:49:18.077646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41640 (local address 127.12.170.124:46349)
0926 13:49:18.077749 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:18.077753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:18.079635 (+ 1882us) server_negotiation.cc:407] Connection header received
0926 13:49:18.079825 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:18.079828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:18.079874 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:18.079919 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:18.080418 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41640: BlockingRecv error: recv got EOF from 127.0.0.1:41640 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 13:49:18.082438 3213 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:49:18.081193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33824 (local address 127.12.170.126:33151)
0926 13:49:18.081311 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:18.081315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:18.081620 (+ 305us) server_negotiation.cc:407] Connection header received
0926 13:49:18.081775 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:18.081778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:18.081825 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:18.081872 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:18.082329 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33824: BlockingRecv error: recv got EOF from 127.0.0.1:33824 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 13:49:18.086262 3289 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:49:18.084895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49222 (local address 127.12.170.125:41597)
0926 13:49:18.084982 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:18.084986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:18.085469 (+ 483us) server_negotiation.cc:407] Connection header received
0926 13:49:18.085649 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:18.085652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:18.085715 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:18.085759 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:18.086175 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49222: BlockingRecv error: recv got EOF from 127.0.0.1:49222 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 13:49:18.090507 3289 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:49:18.089182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49226 (local address 127.12.170.125:41597)
0926 13:49:18.089282 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:18.089285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:18.089733 (+ 448us) server_negotiation.cc:407] Connection header received
0926 13:49:18.089911 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:18.089915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:18.089959 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:18.090003 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:18.090423 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49226: BlockingRecv error: recv got EOF from 127.0.0.1:49226 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260926 13:49:18.095373 3289 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:49:18.094052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49234 (local address 127.12.170.125:41597)
0926 13:49:18.094133 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:18.094136 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:18.094572 (+ 436us) server_negotiation.cc:407] Connection header received
0926 13:49:18.094717 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:18.094720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:18.094760 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:18.094803 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:18.095215 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49234: BlockingRecv error: recv got EOF from 127.0.0.1:49234 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260926 13:49:18.101878 3289 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:49:18.100929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49244 (local address 127.12.170.125:41597)
0926 13:49:18.101043 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:18.101047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:18.101224 (+ 177us) server_negotiation.cc:407] Connection header received
0926 13:49:18.101320 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:18.101323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:18.101360 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:18.101418 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:18.101800 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49244: BlockingRecv error: recv got EOF from 127.0.0.1:49244 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 13:49:18.112246 3289 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:49:18.111331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49252 (local address 127.12.170.125:41597)
0926 13:49:18.111424 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:18.111428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:18.111584 (+ 156us) server_negotiation.cc:407] Connection header received
0926 13:49:18.111709 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:18.111712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:18.111751 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:18.111810 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:18.112163 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49252: BlockingRecv error: recv got EOF from 127.0.0.1:49252 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 13:49:18.130748 3289 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:49:18.129818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49264 (local address 127.12.170.125:41597)
0926 13:49:18.129916 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:18.129920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:18.130095 (+ 175us) server_negotiation.cc:407] Connection header received
0926 13:49:18.130212 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:18.130215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:18.130252 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:18.130308 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:18.130660 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49264: BlockingRecv error: recv got EOF from 127.0.0.1:49264 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 13:49:18.165983 3289 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:49:18.164903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49272 (local address 127.12.170.125:41597)
0926 13:49:18.165054 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:18.165059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:18.165164 (+ 105us) server_negotiation.cc:407] Connection header received
0926 13:49:18.165283 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:18.165286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:18.165348 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:18.165457 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:18.165865 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49272: BlockingRecv error: recv got EOF from 127.0.0.1:49272 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 13:49:18.233039 3289 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:49:18.231949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49274 (local address 127.12.170.125:41597)
0926 13:49:18.232095 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:18.232100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:18.232226 (+ 126us) server_negotiation.cc:407] Connection header received
0926 13:49:18.232359 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:18.232362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:18.232417 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:18.232503 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:18.232922 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49274: BlockingRecv error: recv got EOF from 127.0.0.1:49274 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 13:49:18.364298 3289 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:49:18.363113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49276 (local address 127.12.170.125:41597)
0926 13:49:18.363257 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:18.363262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:18.363440 (+ 178us) server_negotiation.cc:407] Connection header received
0926 13:49:18.363592 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:18.363595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:18.363645 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:18.363733 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:18.364199 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49276: BlockingRecv error: recv got EOF from 127.0.0.1:49276 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 13:49:18.623383 3289 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:49:18.622288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49280 (local address 127.12.170.125:41597)
0926 13:49:18.622472 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:18.622476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:18.622585 (+ 109us) server_negotiation.cc:407] Connection header received
0926 13:49:18.622712 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:18.622714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:18.622763 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:18.622829 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:49:18.623287 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49280: BlockingRecv error: recv got EOF from 127.0.0.1:49280 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 13:49:18.908210 3376 catalog_manager.cc:1259] Loaded cluster ID: 19338c88d84b4cc49e63f1758ccf0f2b
I20260926 13:49:18.908299 3376 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 582f42ac3cb54b83aea0d1ca6205ac0e: loading cluster ID for follower catalog manager: success
I20260926 13:49:18.909989 3376 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 582f42ac3cb54b83aea0d1ca6205ac0e: acquiring CA information for follower catalog manager: success
I20260926 13:49:18.911108 3376 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 582f42ac3cb54b83aea0d1ca6205ac0e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:49:18.940459 3390 catalog_manager.cc:1259] Loaded cluster ID: 19338c88d84b4cc49e63f1758ccf0f2b
I20260926 13:49:18.940533 3390 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e9102e821d8e4e0794fb62493b6c08ad: loading cluster ID for follower catalog manager: success
I20260926 13:49:18.942129 3390 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e9102e821d8e4e0794fb62493b6c08ad: acquiring CA information for follower catalog manager: success
I20260926 13:49:18.943215 3390 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e9102e821d8e4e0794fb62493b6c08ad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:49:19.141129 3487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:19.137515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49288 (local address 127.12.170.125:41597)
0926 13:49:19.137765 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:19.137769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:19.137784 (+ 15us) server_negotiation.cc:407] Connection header received
0926 13:49:19.137927 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:19.137929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:19.138048 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:19.138137 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:49:19.138356 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:19.138882 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:19.139746 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:19.139957 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:19.140022 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:19.140314 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:49:19.140770 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:49:19.140866 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:19.140928 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 13:49:19.140976 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":96,"threads_started":1}
I20260926 13:49:19.146956 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:19.143754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41642 (local address 127.12.170.124:46349)
0926 13:49:19.143964 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:19.143967 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:19.143979 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:49:19.144071 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:19.144074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:19.144176 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:19.144265 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:49:19.144432 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:19.144946 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:19.145739 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:19.145974 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:19.146049 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:19.146266 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:49:19.146637 (+ 371us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:49:19.146719 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:19.146791 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 13:49:19.146835 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":76,"threads_started":1}
I20260926 13:49:19.153262 3487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:19.149478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49296 (local address 127.12.170.125:41597)
0926 13:49:19.149704 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:19.149708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:19.150001 (+ 293us) server_negotiation.cc:407] Connection header received
0926 13:49:19.150167 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:19.150171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:19.150222 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:19.150282 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:49:19.150578 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:19.151106 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:19.152117 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:19.152448 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:19.152510 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:19.152838 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:49:19.152911 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:49:19.152945 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:19.153105 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 13:49:19.153150 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260926 13:49:19.153446 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:19.149932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41654 (local address 127.12.170.124:46349)
0926 13:49:19.150033 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:19.150037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:19.150240 (+ 203us) server_negotiation.cc:407] Connection header received
0926 13:49:19.150328 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:19.150331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:19.150375 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:19.150432 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:49:19.150724 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:19.151453 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:19.152375 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:19.152597 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:19.152688 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:19.152865 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:49:19.152926 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:49:19.152987 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:19.153284 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 13:49:19.153330 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 13:49:19.154150 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:49:19.149507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33828 (local address 127.12.170.126:33151)
0926 13:49:19.149716 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 13:49:19.149719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:49:19.149910 (+ 191us) server_negotiation.cc:407] Connection header received
0926 13:49:19.150128 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:49:19.150131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:49:19.150264 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:49:19.150428 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:49:19.150625 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:19.151432 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:19.152601 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:49:19.152910 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:49:19.153011 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:49:19.153275 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:49:19.153596 (+ 321us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:49:19.153925 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0926 13:49:19.153960 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 13:49:19.154020 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":67,"threads_started":1}
I20260926 13:49:19.156658 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 3141
I20260926 13:49:19.161733 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 3216
I20260926 13:49:19.166594 12969 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0bSs4M/build/debug/bin/kudu with pid 3290
2026-09-26T13:49:19Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2913 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5811 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (267220 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST